Featured FREE Whitepapers

What's New Here?


Applied Big Data : The Freakonomics of Healthcare

I went with a less provocative title this time because my last blog post (http://brianoneill.blogspot.com/2014/04/big-data-fixes-obamacare.html) evidently incited political flame wars. In this post, I hope to avoid that by detailing exactly how Big Data can help our healthcare system in a nonpartisan way. First, let’s decompose the problem a bit. Economics Our healthcare system is still (mostly) based on capitalism: more patients + more visits = more money. Within such a system, it is not in the best interest of healthcare providers to have healthy patients. Admittedly, this is a pessimistic view, and doctors and providers are not always prioritizing financial gain. Minimally however, at a macro-scale there exists a conflict of interest for some segment of the market, because not all healthcare providers profit entirely from preventative care. Behavior Right now, with a few exceptions, everyone pays the same for healthcare. Things are changing, but broadly speaking, there are no financial incentives to make healthy choices. We are responsible only for a fraction of the medical expenses we incur. That means everyone covered by my payer (the entity behind the curtain that actually foots the bills) is helping pay for the medical expenses I may rack up as a result of my Friday night pizza and beer binges. Government Finally, the government is trying. They are trying really hard. Through transparency, reporting, and compliance, they have the correct intentions and ideas to bend the cost curve of healthcare. But the government is the government, and large enterprises are large enterprises. And honestly, getting visibility into the disparate systems of any large single large enterprise is difficult (ask any CIO). Imagine trying to gain visibility into thousands enterprises, all at once. It’s daunting: schematic disparities, messy data, ETL galore. Again, this is a pessimistic view and there are remedies in the works. Things like high deductible plans are making people more aware of their expenses. Payers are trying to transition away from fee-for-service models. (http://en.m.wikipedia.org/wiki/Fee-for-service). But what do these remedies need to be effective? You guessed it. Data. Mounds of it. If you are a payer and want to reward the doctors that are keeping their patients healthy (and out of the doctors offices!), how would you find them? If you are a patient, and want to know who provides the most effective treatments at the cheapest prices, where would you look?  If you are the government and want to know how much pharmaceutical companies are spending on doctors, or which pharmacies are allowing fraudulent prescriptions, what systems would you need to integrate? Hopefully now, you are motivated. This is a big data problem. What’s worse is that it is a messy data problem.  At HMS, its taken us more than three years and significant blood, sweat and tears to put together a platform that deals with the big and messy mound o’ data. The technologies had to mature, along with people and processes. And finally, on sunny days, I can see a light at the end of the tunnel for US healthcare. If you are on the same mission, please don’t hesitate to reach out. Ironically, I’m posting this from a hospital bed as I recover from the bite of a brown recluse spider. I guess there are certain things that big data can’t prevent!Reference: Applied Big Data : The Freakonomics of Healthcare from our JCG partner Brian ONeill at the Brian ONeill’s Blog blog....

Developer Promotion – practical guide & statistics about the Java ecosystem

Before we launched our recent promotion for JavaOne tickets with Java Code Geeks we sought out guides/tutorials about running such a promotion, unfortunately we didn’t find anything useful so we had to improvise and work by instinct. This post will also cover some of the statistical results about the Java community so if this is the reason you are reading this post skip down for some interesting takeaways. Running a promotion tips & tricksThe idea started when one of our talks got accepted to JavaOne, suddenly we had two extra tickets (we got 3 tickets with the booth) so we could give away two tickets. From our experience we came to the conclusion that most developers don’t know about Codename One yet and that is probably the most important thing we need to improve in our outreach. So here are a few conclusions we came to in no particular order:We need a partner to handle the giveaway – this removed the bias from us. We have an incentive to give the tickets to our best customers so we can’t really be fair. That’s why we picked Java Code Geeks. We picked Java Code Geeks as a partner – we needed a partner who is focused on Java (check), who has a big enough audience (check), whose audience isn’t already familiar with our product (check). This is probably the most important step, most of our contestant came from Java Code Geeks and as you can see from the chart to the right. Most of the people who entered our contest didn’t know about Codename One in advance and were happy to find out about us. We created incentives to social share – most of the people who would read Java Code Geeks are probably up to date with the latest technology, but their friends (and friends of friends) might not be. The true value of the promotion is social share which is why we made the contest require sharing. Initially we wanted the person with the most retweets/shares to “win” but Byron from Java Code Geeks objected to this approach (in hindsight I think he was right to object) claiming that it would rule out people who don’t have many friends. It would also have promoted cheating which is a huge problem. We created very detailed rules for the competition – yes, most people didn’t read them (as is obvious by some of the submissions we got) but there are now rules that the raffle itself should follow. We used Google docs to collect details and signups which allowed us to collect a lot of structured data.We included an entirely optional survey and 99% of submitters answered questions within the survey despite the optionality! We promoted the giveaway in our respective newsletters which is usually the most effective way to get people to signup. However, a single retweet by @java is probably more valuable than an 18k strong mailing list. We used Facebook, Twitter and adsense ads – all charged us and delivered literally nothing. Facebook charged for clicks that Google Analytics didn’t show. Normally I would chuck this as a Google analytics omission however we saw a lot of clicks on the Facebook side and no submissions to the contest form… This seems to indicate that Facebook is charging for fraudulent clicks!Results  We asked developers which operating system they were using and were pretty shocked by the results. We see Macs all over JavaOne and all the conferences I go to, but it seems that Macs are a minority to Linux!        The IDE of choice brought a pretty predictable result although much closer than we would have thought as the number of answers grew the distance between the IDE’s grew as well with NetBeans surpassing IDEA and Eclipse growing in strength. However, the race between the IDE’s is still very real and despite Eclipse lead its position seems very assailable in the market. Our decision to support all 3 IDE’s seems like an excellent decision in this light.    Another shocker is the Java platform of choice. I always get the sense that all developers working with Java have migrated to Java EE. This doesn’t seem to be the case entirely, there are plenty of developers still working with Java SE. I’m actually surprised there are still 10% of developers that are working on Java ME.      What is your preferred Java news source, here we excluded Java Code Geeks since we assumed unreasonable bias in the results (according to other stats it would probably occupy well over 50%). Notice that while some outlets (e.g. dzone) are clearly dominant there is no true leader. When targeting the Java community we literally need to be everywhere.    Do Java developers use other languages besides Java? Honestly we should have asked what for? JavaScript/CoffeScript took the lead but do developers use it for HTML or do they use server JavaScript e.g. NodeJS? The main motivation behind asking this question was potential support for other languages in Codename One especially JVM languages e.g. JRuby, Scala etc. We probably should have phrased a few different questions to get refined answers.   Final thoughts I hope you found these results useful and illuminating. For the next contest/survey we will run we picked up the following lessons:Better limits on the goals of the question – e.g. with the languages besides Java we didn’t quite get an answer. I would probably ask, are you interested in other languages? Would you prefer another language? We would not waste a cent on Advertising. Partner selection is critical – Java Code Geeks delivered! As pleased as we were with the work done by the guys at Java  Code Geeks, it was crucial for us to come prepared. We phrased the contest terms and defined most of the conditions/questions. Your partner can be very helpful but only you can define your own expectations and results.Reference: Developer Promotion – practical guide & statistics about the Java ecosystem from our JCG partner Shai Almog at the Codename One blog....

Neo4j: LOAD CSV – Processing hidden arrays in your CSV documents

I was recently asked how to process an ‘array’ of values inside a column in a CSV file using Neo4j’s LOAD CSV tool and although I initially thought this wouldn’t be possible as every cell is treated as a String, Michael showed me a way of working around this which I thought was pretty neat. Let’s say we have a CSV file representing people and their friends. It might look like this:         name,friends "Mark","Michael,Peter" "Michael","Peter,Kenny" "Kenny","Anders,Michael" And what we want is to have the following nodes:Mark Michael Peter Kenny AndersAnd the following friends relationships:Mark -> Michael Mark -> Peter Michael -> Peter Michael -> Kenny Kenny -> Anders Kenny -> MichaelWe’ll start by loading the CSV file and returning each row: $ load csv with headers from "file:/Users/markneedham/Desktop/friends.csv" AS row RETURN row; +------------------------------------------------+ | row | +------------------------------------------------+ | {name -> "Mark", friends -> "Michael,Peter"} | | {name -> "Michael", friends -> "Peter,Kenny"} | | {name -> "Kenny", friends -> "Anders,Michael"} | +------------------------------------------------+ 3 rows As expected the ‘friends’ column is being treated as a String which means we can use the split function to get an array of people that we want to be friends with: $ load csv with headers from "file:/Users/markneedham/Desktop/friends.csv" AS row RETURN row, split(row.friends, ",") AS friends; +-----------------------------------------------------------------------+ | row | friends | +-----------------------------------------------------------------------+ | {name -> "Mark", friends -> "Michael,Peter"} | ["Michael","Peter"] | | {name -> "Michael", friends -> "Peter,Kenny"} | ["Peter","Kenny"] | | {name -> "Kenny", friends -> "Anders,Michael"} | ["Anders","Michael"] | +-----------------------------------------------------------------------+ 3 rows Now that we’ve got them as an array we can use UNWIND to get pairs of friends that we want to create: $ load csv with headers from "file:/Users/markneedham/Desktop/friends.csv" AS row WITH row, split(row.friends, ",") AS friends UNWIND friends AS friend RETURN row.name, friend; +-----------------------+ | row.name | friend | +-----------------------+ | "Mark" | "Michael" | | "Mark" | "Peter" | | "Michael" | "Peter" | | "Michael" | "Kenny" | | "Kenny" | "Anders" | | "Kenny" | "Michael" | +-----------------------+ 6 rows And now we’ll introduce some MERGE statements to create the appropriate nodes and relationships: $ load csv with headers from "file:/Users/markneedham/Desktop/friends.csv" AS row WITH row, split(row.friends, ",") AS friends UNWIND friends AS friend MERGE (p1:Person {name: row.name}) MERGE (p2:Person {name: friend}) MERGE (p1)-[:FRIENDS_WITH]->(p2); +-------------------+ | No data returned. | +-------------------+ Nodes created: 5 Relationships created: 6 Properties set: 5 Labels added: 5 373 ms And now if we query the database to get back all the nodes + relationships… $ match (p1:Person)-[r]->(p2) RETURN p1,r, p2; +------------------------------------------------------------------------+ | p1 | r | p2 | +------------------------------------------------------------------------+ | Node[0]{name:"Mark"} | :FRIENDS_WITH[0]{} | Node[1]{name:"Michael"} | | Node[0]{name:"Mark"} | :FRIENDS_WITH[1]{} | Node[2]{name:"Peter"} | | Node[1]{name:"Michael"} | :FRIENDS_WITH[2]{} | Node[2]{name:"Peter"} | | Node[1]{name:"Michael"} | :FRIENDS_WITH[3]{} | Node[3]{name:"Kenny"} | | Node[3]{name:"Kenny"} | :FRIENDS_WITH[4]{} | Node[4]{name:"Anders"} | | Node[3]{name:"Kenny"} | :FRIENDS_WITH[5]{} | Node[1]{name:"Michael"} | +------------------------------------------------------------------------+ 6 rows …you’ll see that we have everything. If instead of a comma separated list of people we have a literal array in the cell… name,friends "Mark", "[Michael,Peter]" "Michael", "[Peter,Kenny]" "Kenny", "[Anders,Michael]" …we’d need to tweak the part of the query which extracts our friends to strip off the first and last characters: $ load csv with headers from "file:/Users/markneedham/Desktop/friendsa.csv" AS row RETURN row, split(substring(row.friends, 1, length(row.friends) -2), ",") AS friends; +-------------------------------------------------------------------------+ | row | friends | +-------------------------------------------------------------------------+ | {name -> "Mark", friends -> "[Michael,Peter]"} | ["Michael","Peter"] | | {name -> "Michael", friends -> "[Peter,Kenny]"} | ["Peter","Kenny"] | | {name -> "Kenny", friends -> "[Anders,Michael]"} | ["Anders","Michael"] | +-------------------------------------------------------------------------+ 3 rows And then if we put the whole query together we end up with this: $ load csv with headers from "file:/Users/markneedham/Desktop/friendsa.csv" AS row WITH row, split(substring(row.friends, 1, length(row.friends) -2), ",") AS friends UNWIND friends AS friend MERGE (p1:Person {name: row.name}) MERGE (p2:Person {name: friend}) MERGE (p1)-[:FRIENDS_WITH]->(p2);; +-------------------+ | No data returned. | +-------------------+ Nodes created: 5 Relationships created: 6 Properties set: 5 Labels added: 5Reference: Neo4j: LOAD CSV – Processing hidden arrays in your CSV documents from our JCG partner Mark Needham at the Mark Needham Blog blog....

Is TDD Dead Or Alive?

This blog post is for those of you who are unaware that there is a major debate in contemporary software development happening now, today. People have been wondering about the value of Test-Driven Development (TDD) for a long while, but it was not until David Heinemeier Hansson of 37 Signals  posted a blog article on St. George’s Day (23rd April 2014 and a significant day personally for me) called “TDD is Dead. Long live testing”. It started an unstoppable momentum of discussion into what is now a heated and controversial topic. So I encourage you read his article, first, if you are an experienced software engineer and then watch the videos. After this article was posted to the Internet, Heinemeier Hanson, Martin Fowler and Kent Beck followed up with a series of Google Hangout recordings, in May and June, each one was about 30 minutes long. The last edition (part V and VII), which was full hour long was very revealing, because it was a question and answer sessions from the community and therefore extended the discussion to the wider audience. If you have 3 hours of free time, instead watching another Hollywood movie or re-runs of World Cup football, then it is well worth your time to listening (and/or watching) the entire series to understand the debate and different points of view. As for my own opinion on TDD, it has a place in my toolbox like another hammer or a screwdriver, and it should be treated as such. To be a TDD practitioner requires skill and discipline; and the knowledge to believe that it is not appropriate in all situations. I have some sympathy with Heinemeier Hansson’s frustrated view, “Most people cannot just leave good ideas the f**k alone”. I witnessed a certain zealotry in a couple job interviews a few years ago, when interviewers used the technique like an officer using a truncheon to beat somebody with. If you didn’t write code in a correct view for the client, you were rejected. I raised suspicions in my head about the variations of different testing styles across different organisations and sectors. In the end, I have suspected that you do not have write code test first all the time, with the plethora of unit testing frameworks out there such JUnit, ScalaTest, Scala Specs. It is more important to have the ground level, there must be tests in the application and system software that exercise the customer’s acceptance requirement and produces a Minimal Viable Product. This is definite the road to follow. PS: The first video in the series, “Is TDD Dead?”, starts here.Reference: Is TDD Dead Or Alive? from our JCG partner Peter Pilgrim at the Peter Pilgrim’s blog blog....

10 things you can do to as a developer to make your app secure: #10 Design Security In

There’s more to secure design and architecture besides properly implementing Authentication, Access Control and Logging strategies, and choosing (and properly using) a good framework. You need to consider and deal with security threats and risks at many different points in your design. Adam Shostack’s new book on Threat Modeling explores how to do this in detail, with lots of exercises and examples on how to look for and plug security holes in software design, and how to think about design risks.   But some important basic ideas in secure design will take you far: Know your Tools When deciding on the language(s) and technology stack for the system, make sure that you understand the security constraints and risks that your choices will dictate. If you’re using a new language, take time to learn about how to write code properly and safely in that language. If you’re programming in Java, C or C++ or Perl check out CERT’s secure coding guidelines for those languages. If you’re writing code on iOS, read Apple’s Secure Coding Guide. For .NET, review OWASP’s .NET Security project. Look for static analysis tools like Findbugs and PMD for Java, JSHint for Javascript, OCLint for C/C++ and Objective-C, Brakeman for Ruby, RIPS for PHP, Microsoft’s static analysis tools for .NET, or commercial tools that will help catch common security bugs and logic bugs in coding or Continuous Integration. And make sure that you (or ops) understand how to lock down or harden the O/S and to safely configure your container and database (or NoSQL data) manager. Tiering and Trust Tiering or layering, and trust in design are closely tied together. You must understand and verify trust assumptions at the boundaries of each layer in the architecture and between systems and between components in design, in order to decide what security controls need to be enforced at these boundaries: authentication, access control, data validation and encoding, encryption, logging. Understand when data or control crosses a trust boundary: to/from code that is outside of your direct control. This could be an outside system, or a browser or mobile client or other type of client, or another layer of the architecture or another component or service. Thinking about trust is much simpler and more concrete than thinking about threats. And easier to test and verify. Just ask some simple questions: Where is the data coming from? How can you be sure? Can you trust this data – has it been validated and safely encoded? Can you trust the code on the other side to protect the integrity and confidentiality of data that you pass to it? Do you know what happens if an exception or error occurs – could you lose data or data integrity, or leak data, does the code fail open or fail closed? Before you make changes to the design, make sure that you understand these assumptions and make sure that the assumptions are correct. The Application Attack Surface Finally, it’s important to understand and manage the system’s Attack Surface: all of the ways that attackers can get in, or get data out, of the system, all of the resources that are exposed to attackers. APIs, files, sockets, forms, fields, URLs, parameters, cookies. And the security plumbing that protects these parts of the system. Your goal should be to try to keep the Attack Surface as small as possible. But this is much easier said than done: each new feature and new integration point expands the Attack Surface. Try to understand the risks that you are introducing, and how serious they are. Are you creating a brand new network-facing API or designing a new business workflow that deals with money or confidential data, or changing your access control model, or swapping out an important part of your platform architecture? Or are you just adding yet another CRUD admin form, or just one more field to an existing form or file. In each case you are changing the Attack Surface, but the risks will be much different, and so will the way that you need to manage these risks. For small, well-understood changes the risks are usually negligible – just keep coding. If the risks are high enough you’ll need to do some abuse case analysis or threat modeling, or make time for a code review or pen testing. And of course, once a feature or option or interface is no longer needed, remove it and delete the code. This will reduce the system’s Attack Surface, as well as simplifying your maintenance and testing work. That’s it. We’re done. The 10 things you can do as a developer to make your app secure: from thinking about security in architectural layering and technology choices, including security in requirements, taking advantage of other people’s code by using frameworks and libraries carefully, making sure that you implement basic security controls and features like Authentication and Access Control properly, protecting data privacy, logging with security in mind, and dealing with input data and stopping injection attacks, especially SQL injection.   This isn’t an exhaustive list. But understanding and dealing with these important issues in application security – including security when you think about requirements and design and coding and testing, knowing more about your tools and using them properly – is work that all developers can do, and will take you a long, long way towards making your system secure and reliable.Reference: 10 things you can do to as a developer to make your app secure: #10 Design Security In from our JCG partner Jim Bird at the Building Real Software blog....

JavaFX Tip 5: Be Observable

Even in this time of total NSA surveillance it is still a good idea to implement your JavaFX controls with observability in mind. This is easy to achieve in JavaFX especially compared to Swing. The Old Days Coming from Swing I was used to spending a lot of energy and time on making custom controls observable. It usually required adding methods to add and remove listeners of a certain type. This listener type was a new interface, the single method of that interface accepted a new event object. To send this event object the control had to “fire” it inside my custom control, so I ended up writing fireXYZ() methods. A lot of work only to let somebody know that some aspect of the control’s state has changed. The New Age In JavaFX observability can be achieved much more easily by using properties and observable collections. In the FlexGanttFX framework almost every attribute of its controls are properties and all collections are created via the FXCollections class, which returns observable collections. This makes a lot of sense because the total of the attributes of each control define its state and this state needs to be observable so that other parts of the application (especially other controls) can react to state changes. Boilerplate Code The downside of using properties is that you end up writing a lot of boilerplate code but still much less than in Swing. In the case of a boolean property it will look like this: public final BooleanProperty showStuff = new SimpleBooleanProperty(this, "showStuff", true);public final BooleanProperty showStuffProperty() { return showStuff; }public final boolean isShowStuff() { return showStuff.get(); }public final void setShowStuff(boolean show) { showStuff.set(show); } And yes, you should follow exactly this coding convention. For a property called xxx provide a property accessor method called xxxProperty(), a setter called setXXX() and a getter called getXXX(). This is the way it is used throughout JavaFX and you should stick to it. When working with collections you typically end up with at least this code: private final ObservableList<String> names = FXCollections.observableArrayList();public final ObservableList<String> getNames() { return names; } In the JavaFX core controls collections are often stored in a property so that the whole collection can be replaced by a new collection. So far I haven’t really seen a strong requirement for this in my controls and not doing so means I do’t have to write property listeners for this case. Listeners Now that our control attributes are properties we can easily react to state changes by adding listeners to them (using lambda expressions). myControl.showStuffProperty(). addListener(it -> showStuffChanged());myControl.getNames(). addListener((Observable observable) -> namesChanged()); Conclusion When designing your own custom control expose its state in the control class via properties and observable collections. By doing so everyone will be able to observe the state of your controls in a way that is consistent with the entire JavaFX framework and that will enable other developers to seamlessly integrate their controls / their application with your controls.Reference: JAVAFX TIP 5: BE OBSERVABLE from our JCG partner Dirk Lemmermann at the Pixel Perfect blog....

JavaFX Tip 4: Have The Final Word

When developing custom controls for JavaFX I would highly recommend to follow in the footsteps of  the core JavaFX controls and to make the API of your controls as final as possible and to put the “final” keyword in front of all your method declarations. Example In FlexGanttFX I have a lot of code that looks like this:         public final Activity getActivityAt(double x, double y) { ... } public final Row getRowAt(double y) { ... } public final void setShowLinks(boolean show) { ... } Why? When you design a control you have a specific behaviour of the control in mind. This behaviour can be protected by using “final”. If you do not use it then other developers will subclass the control, they will override its methods, they will run into problems, and they will submit a defect ticket to your issue management tool. In the end you will waste your time on chasing their bugs. Bugs that only exist because your control was used in a way that you couldn’t possibly foresee. Where? Protection is especially needed for the public API of the control class, which is managing the “state” of the control. It is the contract between you and the user of your control. It is less true for the skin of the control, because skins are “implementation detail” and people modifying skins know that they are interfering with the inner workings of the control. You still want to make it easy for other developers to modify the “appearance” of the control. New Concept? Protecting your controls this way is not a new concept but it wasn’t used very much in Swing. The case for using “final” is stronger now because JavaFX controls are much more observable than Swing controls. They are exposing their state via properties and observable collections. This way applications can react to state changes by “listening” to the control instead of plugging themself into the control by overriding its methods. Another reason is the clean separation into “control” and “skin” in JavaFX. Swing components often had “skin” aspects right in the component itself and not in the UI delegate (e.g. paintComponent()).Reference: JAVAFX TIP 4: HAVE THE FINAL WORD from our JCG partner Dirk Lemmermann at the Pixel Perfect blog....

The madness of layered architecture

I once visited a team that had fifteen layers in their code. That is: If you wanted to display some data in the database in a web page, that data passed through 15 classes in the application. What did these layers do? Oh, nothing much. They just copied data from one object to the next. Or sometimes the “access object layer” would perform a check that objects were valid. Or perhaps the check would be done in the “boundary object layer”. It varied, depending on which part of the application you looked. Puzzled (and somewhat annoyed), I asked the team why they had constructed their application this way. The answer was simple enough: They had been told so by the expensive consultant who had been hired to advice on the organization’s architecture. I asked the team what rationale the consultant had given. They just shrugged. Who knows? Today, I often visit teams who have three to five layers in their code. When asked why, the response is usually the same: This is the advice they have been given. From a book, a video or a conference talk. And the rationale remains elusive or muddled at best. Why do we construct layered applications? There’s an ancient saying in the field of computing: Any problem in computer science can be solved by adding a layer of indirection. Famously, this is the guiding principle behind our modern network stack. In web services SOAP performs method calls on top of HTTP. HTTP sends requests and receives responses on top of TCP. TCP streams data in two directions on top of IP. IP routes packets of bits through a network on top of physical protocols like Ethernet. Ethernet broadcasts packets of bits with a destination address to all computers on a bus. Each layer performs a function that lets the higher layer abstract away the complexities of for example resending lost packets or routing packets through a globally interconnected network. The analogy is used to argue for layers in enterprise application architecture. But enterprise applications are not like network protocols. Every layer in most enterprise application operates at the same level of abstraction. To pick on a popular example: John Papa’s video on Single Page Applications uses the following layers on the server side (and a separate set on the client side): Controllers, UnitOfWork, Repository, Factories and EntityFramework. So for example the AttendanceRepository property in CodeCamperUnitOfWork returns a AttendanceRepository to the AttendanceController, which calls GetBySessionId() method in AttendanceRepository layer, which finally calls DbSet.Where(ps => ps.SessionId == sessionId) on EntityFramework. And then there’s the RepositoryFactories layers. Whee! And what does it all do? It filters an entity based on a parameter. Wat?! (A hint that this is going off the rails is that discussion in the video presentation starts with the bottom and builds up to the controllers instead of outside in) In a similar Java application, I have seen – and feel free to skip these tedious details – the SpeakersController.findByConference calls SpeakersService.findByConference, which calls SpeakersManager.findByConference, which calls SpeakersRepository.findByConference, which constructs a horrific JPAQL query which nobody can understand. JPA returns an @Entity which is mapped to the database, and the Repository, or perhaps the Manager, Service or Controller, or perhaps two or three of these, will transform from Speaker-class to another. Why is this a problem? The cost of code: A reasonable conjecture would be that the cost of developing and maintaining an application grows with the size of the application. Adding code without value is waste. Single responsibility principle: In the above example, the SpeakerService will often contain all functionality associated with speakers. So if adding a speaker requires you to select a conference from a drop-down list, the SpeakerService will often have a findAllConferences method, so that SpeakersController doesn’t need to also have a dependency on ConferenceService. However, this makes the classes into functionality magnets. The symptom is low coherence: the methods of one class can be divided into distinct sets that are never used at the same time. Dumb services: “Service” is a horrible name for a class – a service is a more or less coherent collection of functions. A more meaningful name would be a “repository” for a service that stores and retrieves objects, a Query is a service that selects objects based on a criteria (actually it’s a command, not a service), a Gateway is a service that communicates with another system, a ReportGenerator is a service that creates a report. Of course, the fact that a controller may have references to a repository, a report generator and a gateway should be quite normal if the controller fetches data from the database to generate a report for another system. Multiple points of extension: If you have a controller that calls a service that calls a manager that calls a repository and you want to add some validation that the object you are saving is consistent, where would you add it? How much would you be willing to bet that the other developers on the team would give the same answer? How much would you be willing to bet that you would give the same answer in a few months? Catering to the least common denominator: In the conference application we have been playing with, DaysController creates and returns the days available for a conference. The functionality needed for DaysController is dead simple. On the other hand TalksController has a lot more functionality. Even though these controllers have vastly different needs, they both get the same (boring) set of classes: A Controller, a UnitOfWork, a Repository. There is no reason the DaysController couldn’t use EntityFramework directly, other than the desire for consistency. Most applications have a few functional verticals that contain the meat of the application and a lot of small supporting verticals. Treating them the same only creates more work and more maintenance effort. So how can you fix it? The first thing you must do is to build your application from the outside in. If your job is to return a set of objects, with .NET EntityFramework you can access the DbSet directly – just inject IDbSet in your controller. With Java JPA, you probably want a Repository with a finder method to hide the JPAQL madness. No service, manager, worker, or whatever is needed. The second thing you must do is to grow your architecture. When you realize that there’s more responsibilities in your controller than deciding what to do with a user request, you must extract new classes. You may for example need a PdfScheduleGenerator to create a printable schedule for your conference. If you’re using .NET entity framework, you many want to create some LINQ extension methods on e.g. IEnumerable(which is extended by IDbSet). The third and most important thing you must do is to give your classes names that reflect their responsibilities. A service should not just be a place to dump a lot of methods. Every problem in computer science can be solved by adding a layer of indirection, but most problems in software engineering can be solved by removing a misplaced layer. Let’s build leaner applications!Reference: The madness of layered architecture from our JCG partner Johannes Brodwall at the Thinking Inside a Bigger Box blog....

Test Attributes #1: Validity

In my last post, I created a list of test attributes. If one of them isn’t ok, you need to do some fixin’. This is the first of a series of posts that is going to discuss the different faces of tests. Let’s start with validity. Admittedly, it’s not the first of attribute I thought about. What are the chances we’re going to write a wrong test?   How can this happen? We usually write tests based on our understanding of the code, the requirements we need to implement, and we fill the gap by assumptions. We can be wrong on either. Or all. A more interesting question is: How do we know we’ve written an incorrect test? We find out we have the wrong tests in one or more ways:The Review: Someone looks at our test and code, and tells us we’re either testing the wrong thing, or that it isn’t the way to prove our code works. The Surprise: Our test surprisingly fails, where it should have passed. It can happen the other way too. The Smell: We have a feeling we’re on the wrong path. The test passes, but something feels wrong. The Facepalm: A penny drops, and we get  the “what the hell are you doing?” followed by “glad no one noticed” feelings.Once we know what is wrong we can easily fix it. But wouldn’t it be easier if we avoided this altogether? The easiest way is to involve someone. Pair programming helps avoid bugs and writing the wrong test. Obviously, in the Review a partner helps, but they can also avoid the Surprise, Smell and the Facepalm. And you don’t want them there when you have a Facepalm moment, right? One of the foundation of agile development is feedback. We all make mistakes. In agile development we acknowledge that, so we put brakes in place, such as pair programming, to identify problems early. Next up: Readability.Reference: Test Attributes #1: Validity from our JCG partner Gil Zilberfeld at the Geek Out of Water blog....

Compounding double error

Overview In a previous article, I outlined why BigDecimal is not the answer most of the time. While it is possible to construct situations where double produces an error, it is also just as easy to construct situations were BigDecimal get an error. BigDecimal is easier to get right, but easier to get wrong. The anecdotal evidence is that junior developers don’t have as much trouble getting BigDecimal right as they do getting double with rounding right.  However, I am sceptical of this because in BigDecimal it is much easier for an error to go unnoticed as well. Lets take this example where double produces an incorrect answer. double d = 1.00; d /= 49; d *= 49 * 2; System.out.println("d=" + d);BigDecimal bd = BigDecimal.ONE; bd = bd .divide(BigDecimal.valueOf(49), 2, BigDecimal.ROUND_HALF_UP); bd = bd.multiply(BigDecimal.valueOf(49*2)); System.out.println("bd=" + bd); printsd=1.9999999999999998 bd=1.96In this case, double looks wrong, it needs rounding which would give the correct answer of 2.0. However the BigDecimal looks right, but it isn’t due to representation error. We could change the division to use more precision, but you will always get a representation error, though you can control how small that error is. You have to ensure numbers are real and use rounding. Even with BigDecimal, you have to use appropriate rounding. Lets say you have a loan for $1,000,000 and you apply 0.0005% interest per day. The account can only have a whole number of cents, so rounding is needed to make this a real amount of money. If don’t do this how long does it take to make a 1 cent difference?double interest = 0.0005; BigDecimal interestBD = BigDecimal.valueOf(interest);double amount = 1e6; BigDecimal amountBD = BigDecimal.valueOf(amount); BigDecimal amountBD2 = BigDecimal.valueOf(amount);long i = 0; do { System.out.printf("%,d: BigDecimal: $%s, BigDecimal: $%s%n", i, amountBD, amountBD2); i++; amountBD = amountBD.add(amountBD.multiply(interestBD) .setScale(2, BigDecimal.ROUND_HALF_UP)); amountBD2 = amountBD2.add(amountBD2.multiply(interestBD));} while (amountBD2.subtract(amountBD).abs() .compareTo(BigDecimal.valueOf(0.01)) < 0); System.out.printf("After %,d iterations the error was 1 cent and you owe %s%n", i, amountBD);prints finally8: BigDecimal: $1004007.00, BigDecimal: $1004007.00700437675043756250390625000000000000000 After 9 iterations the error was 1 cent and you owe 1004509.00You could round the result but this hide the fact you are off by a cent even though you used BigDecimal. double eventually has a representation error Even if you use appropriate rounding, double will give you an incorrect result. It is much later than the previous example. double interest = 0.0005; BigDecimal interestBD = BigDecimal.valueOf(interest); double amount = 1e6; BigDecimal amountBD = BigDecimal.valueOf(amount); long i = 0; do { System.out.printf("%,d: double: $%.2f, BigDecimal: $%s%n", i, amount, amountBD); i++; amount = round2(amount + amount * interest); amountBD = amountBD.add(amountBD.multiply(interestBD) .setScale(2, BigDecimal.ROUND_HALF_UP)); } while (BigDecimal.valueOf(amount).subtract(amountBD).abs() .compareTo(BigDecimal.valueOf(0.01)) < 0); System.out.printf("After %,d iterations the error was 1 cent and you owe %s%n", i, amountBD); prints finally22,473: double: $75636308370.01, BigDecimal: $75636308370.01 After 22,474 iterations the error was 1 cent and you owe 75674126524.20From an IT perspective we have an error of one cent, from a business perspective we have a client who has made no repayments for more than 9 years and owes the bank $75.6 billion, enough to bring down the bank. If only the IT guy had used BigDecimal!? Conclusion My final recommendation is that you should use what you feel comfortable with, don't forget about rounding, do use real numbers, not whatever the mathematics produces e.g. can I have fractions of a cent, or can I trade fractions of share.  Don't forget about the business perspective. You might find that BigDecimal makes more sense for your company, your project or your team. Don't assume BigDecimal is the only way, don't assume the problems double faces don't apply also to BigDecimal. BigDecimal is not a ticket to best practice coding, because complacency is a sure way of introducing errors.Reference: Compounding double error from our JCG partner Peter Lawrey at the Vanilla Java blog....
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books