Featured FREE Whitepapers

What's New Here?

java-logo

Creating Your Own Java Annotations

If you’ve been programming in Java and using any one of the popular frameworks like Spring and Hibernate, you should be very familiar with using annotations. When working with an existing framework, its annotations typically suffice. But, have you ever found a need to create your own annotations? Not too long ago, I found a reason to create my own annotations for a project that involved verifying common data stored in multiple databases.         The Scenario The business had multiple databases that were storing the same information and had various means of keeping the data up to date. The business had planned a project to consolidate the data into a master database to alleviate some of the issues involved with having multiple sources of data. Before the project could begin however, the business needed to know how far out of sync the data was and make any necessary corrections to get in back in sync. The first step required creating a report that showed common data that belonged in multiple databases and validated the values, highlighting any records that didn’t match according to the reconciliation rules defined. Here’s a short summary of the requirements at the time:Compare the data between multiple databases for a common piece of data, such as a customer, company, or catalog information. By default the value found should match exactly across all of the databases based upon the type of value. For certain fields we only want to display the value found and not perform any data comparison. For certain fields we only want to compare the value found and perform data verification on the specific data sources specified. For certain fields we may want to do some complicated data comparisons that may be based on the value of other fields within the record. For certain fields we may want to format the data in a specific format, such as $000,000.00 for monetary amounts. The report should be in MS Excel format, each row containing the field value from each source. Any row that doesn’t match according to the data verification rules should be highlighted in yellow.Annotations After going over the requirements and knocking around a few ideas, I decided to use annotations to drive the configuration for the data comparison and reporting process. We needed something that was somewhat simple, yet highly flexible and extensible. These annotations will be at the field level and I like the fact that the configuration won’t be hidden away in a file somewhere on the classpath. Instead you’ll be able to look at the annotation associated with a field to know exactly how it will be processed. In the simplest terms, an annotation is nothing more than a marker, metadata that provides information but has no direct effect on the operation of the code itself. If you’ve been doing Java programming for a while now you should be pretty familiar with their use, but maybe you’ve never had a need to create your own. To do that you’ll need to create a new type that uses the Java type @interface that will contain the elements that specify the details of the metadata. Here’s an example from the project: @Target(ElementType.FIELD) @Retention(RetentionPolicy.RUNTIME) public @interface ReconField {/** * Value indicates whether or not the values from the specified sources should be compared or will be used to display values or reference within a rule. * * @return The value if sources should be compared, defaults to true. */ boolean compareSources() default true;/** * Value indicates the format that should be used to display the value in the report. * * @return The format specified, defaulting to native. */ ReconDisplayFormat displayFormat() default ReconDisplayFormat.NATIVE;/** * Value indicates the ID value of the field used for matching source values up to the field. * * @return The ID of the field. */ String id();/** * Value indicates the label that should be displayed in the report for the field. * * @return The label value specified, defaults to an empty string. */ String label() default "";/** * Value that indicates the sources that should be compared for differences. * * @return The list of sources for comparison. */ ReconSource[] sourcesToCompare() default {};} This is the main annotation that will drive how the data comparison process will work. It contains the basic elements required to fulfill most of the requirements for comparing the data amongst the different data sources. The @ReconField should handle most of what we need except for the requirement of more complex data comparison, which we’ll go over a little bit later. Most of these elements are explained by the comments associated with each one in the code listing, however there are a couple of key annotations on our @ReconField that need to be pointed out.@Target – This annotation allows you to specify which java elements your annotation should apply to. The possible target types are ANNOTATION_TYPE, CONSTRUCTOR, FIELD, LOCAL_VARIABLE, METHOD, PACKAGE, PARAMETER and TYPE. In our @ReconField annotation it is specific to the FIELD level. @Retention – This allows you to specify when the annotation will be available. The possible values are CLASS, RUNTIME and SOURCE. Since we’ll be processing this annotation at RUNTIME, that’s what this needs to be set to.This data verification process will run one query for each database and then map the results to a common data bean that represents all of the fields for that particular type of business record. The annotations on each field of this mapped data bean tell the processor how to perform the data comparison for that particular field and its value found on each database. So let’s look at a few examples of how these annotations would be used for various data comparison configurations. To verify that the value exists and matches exactly in each data source, you would only need to provide the field ID and the label that should be displayed for the field on the report. @ReconField(id = CUSTOMER_ID, label = "Customer ID") private String customerId; To display the values found in each data source, but not do any data comparisons, you would need to specify the element compareSources and set its value to false. @ReconField(id = NAME, label = "NAME", compareSources = false) private String name; To verify the values found in specific data sources but not all of them, you would use the element sourcesToCompare. Using this would display all of the values found, but only perform any data comparisons on the data sources listed in the element. The handles the case in which some data is not stored in every data source. ReconSource is an enum that contains the data sources available for comparison. @ReconField(id = PRIVATE_PLACEMENT_FLAG, label = "PRIVATE PLACEMENT FLAG", sourcesToCompare ={ ReconSource.LEGACY, ReconSource.PACE }) private String privatePlacementFlag; Now that we’ve covered our basic requirements, we need to address the ability to run complex data comparisons that are specific to the field in question. To do that, we’ll create a second annotation that will drive the processing of custom rules. @Target(ElementType.FIELD) @Retention(RetentionPolicy.RUNTIME) public @interface ReconCustomRule {/** * Value indicates the parameters used to instantiate a custom rule processor, the default value is no parameters. * * @return The String[] of parameters to instantiate a custom rule processor. */ String[] params() default {};/** * Value indicates the class of the custom rule processor to be used in comparing the values from each source. * * @return The class of the custom rule processor. */ Class<?> processor() default DefaultReconRule.class;} Very similar to the previous annotation, the biggest difference in the @ReconCustomRule annotation is that we are specifying a class that will execute the data comparison when the recon process executes. You can only define the class that will be used, so your processor will have to instantiate and initialize any class that you specify. The class that is specified in this annotation will need to implement a custom rule interface, which will be used by the rule processor to execute the rule. Now let’s take a look at a couple of examples of this annotation. In this example, we’re using a custom rule that will check to see if the stock exchange is not the United States and skip the data comparison if that’s the case. To do this, the rule will need to check the exchange country field on the same record. @ReconField(id = STREET_CUSIP, label = "STREET CUSIP", compareSources = false) @ReconCustomRule(processor = SkipNonUSExchangeComparisonRule.class) private String streetCusip; Here’s an example where we are specifying a parameter for the custom rule, in this case it’s a tolerance amount. For this specific data comparison, the values being compared cannot be off by more than 1,000. By using a parameter to specify the tolerance amount, this allows us to use the same custom rule on multiple fields with different tolerance amounts. The only drawback is that these parameters are static and can’t be dynamic due to the nature of annotations. @ReconField(id = USD_MKT_CAP, label = "MARKET CAP USD", displayFormat = ReconDisplayFormat.NUMERIC_WHOLE, sourcesToCompare = { ReconSource.LEGACY, ReconSource.PACE, ReconSource.BOB_PRCM }) @ReconCustomRule(processor = ToleranceAmountRule.class, params = { "10000" }) private BigDecimal usdMktCap; As you can see, we’ve designed quite of bit of flexibility into a data verification report for multiple databases by just using a couple of fairly simple annotations. For this particular case, the annotations are driving the data comparison processing so we’re actually evaluating the annotations that we find on the mapped data bean and using those to direct the processing. Conclusion There are numerous articles out there already about Java annotations, what they do, and the rules for using them. I wanted this article to focus more on an example of why you might want to consider using them and see the benefit directly. Keep in mind that this is only the starting point, once you have decided on creating annotations you’ll still need to figure out how to process them to really take full advantage of them. In part two, I’ll show you how to process these annotations using Java reflection. Until then, here are a couple of good resources to learn more about Java annotations:The Java Annotation Tutorial - http://docs.oracle.com/javase/tutorial/java/annotations/ Java Annotations - http://tutorials.jenkov.com/java/annotations.html How Annotations Work - http://java.dzone.com/articles/how-annotations-work-javaReference: Creating Your Own Java Annotations from our JCG partner Jonny Hackett at the Keyhole Software blog....
json-logo

Converting JSON to XML to Java Objects using XStream

XStream library can be an effective tool for converting JSON to Java to XML translations to and fro. Lets explore each one of them one by one, and see which driver is used. Handling JSONs To convert JSON to Java objects all you have to do is initialize XStream object with an appropriate driver and you are ready to serialise your objects to (and from) JSON. XStream currently delivers two drivers for JSON to Object ocnversion:  JsonHierarchicalStreamDriver: This does not have an additional dependency, but can only be used to write XMLJettisonMappedXmlDriver: This is based on Jettison and can also deserialize JSON to Java objects again.Jettison driver Jettison driver uses Jettison StAX parser to read and write data in JSON format. It is available in XStream since version 1.2.2 and is implemented in com.thoughtworks.xstream.io.json.JettisonMappedXmlDriver class. In order to get this working, we need to add the dependencies in pom : <dependencies> <dependency> <groupId>com.thoughtworks.xstream</groupId> <artifactId>xstream</artifactId> <version>1.4.7</version> </dependency> <dependency> <groupId>org.codehaus.jettison</groupId> <artifactId>jettison</artifactId> <version>1.1</version> </dependency> </dependencies> And the code to convert JSON to object and object to Json : XStream xstream = new XStream(new JettisonMappedXmlDriver()); xstream.toXML(xml); //converts Object to JSON xstream.fromXML(obj); //Converts Json to Object Serializing an object to XML To serialize an Object to XML XStream uses 2 drivers :StaxDriver XStream xstream = new XStream(new StaxDriver()); xstream.toXML(xml); //converts Object to XML xstream.fromXML(obj); //Converts XML to ObjectDomDriver XStream xstream = new XStream(new DomDriver()); xstream.toXML(xml); //converts Object to XML xstream.fromXML(obj); //Converts XML to ObjectFinally, lets see all these in one class: package com.anirudh;import com.thoughtworks.xstream.XStream; import com.thoughtworks.xstream.io.json.JettisonMappedXmlDriver; import com.thoughtworks.xstream.io.xml.DomDriver; import com.thoughtworks.xstream.io.xml.StaxDriver;/** * Created by anirudh on 15/07/14. */ public class Transformer<T> {private static final XStream XSTREAM_INSTANCE = null;public T getObjectFromJSON(String json){ return (T) getInstance().fromXML(json); }public String getJSONFromObject(T t){ return getInstance().toXML(t); }private XStream getInstance(){ if(XSTREAM_INSTANCE==null){ return new XStream(new JettisonMappedXmlDriver()); } else { return XSTREAM_INSTANCE; } }public T getObjectFromXML(String xml){ return (T)getStaxDriverInstance().fromXML(xml); }public String getXMLFromObject(T t){ return getStaxDriverInstance().toXML(t); }public T getObjectFromXMLUsingDomDriver(String xml){ return (T)getDomDriverInstance().fromXML(xml); }public String getXMLFromObjectUsingDomDriver(T t){ return getDomDriverInstance().toXML(t); }private XStream getStaxDriverInstance(){ if(XSTREAM_INSTANCE==null) { return new XStream(new StaxDriver()); }else{ return XSTREAM_INSTANCE; } }private XStream getDomDriverInstance(){ if(XSTREAM_INSTANCE==null){ return new XStream(new DomDriver()); }else{ return XSTREAM_INSTANCE; } } } Write a JUnit class to test it: package com.anirudh;import com.anirudh.domain.Product; import org.junit.Before; import org.junit.Test;/** * Created by anirudh on 15/07/14. */ public class TransformerTest {private Transformer<Product> productTransformer; private Product product; @Before public void init(){ productTransformer = new Transformer<Product>(); product = new Product(123,"Banana",23.00); } @Test public void testJSONToObject(){ String json = productTransformer.getJSONFromObject(product); System.out.println(json); Product convertedproduct = productTransformer.getObjectFromJSON(json); System.out.println(convertedproduct.getName()); }@Test public void testXMLtoObjectForStax(){ String xml = productTransformer.getXMLFromObject(product); System.out.println(xml); Product convertedproduct = productTransformer.getObjectFromXML(xml); System.out.println(convertedproduct.getName()); }@Test public void testXMLtoObjectForDom(){ String xml = productTransformer.getXMLFromObjectUsingDomDriver(product); System.out.println(xml); Product convertedproduct = productTransformer.getObjectFromXMLUsingDomDriver(xml); System.out.println(convertedproduct.getName()); }} The full code can be seen here. In the next blog, we will compare the use cases, exploring where what fits in.Reference: Converting JSON to XML to Java Objects using XStream from our JCG partner Anirudh Bhatnagar at the anirudh bhatnagar blog....
agile-logo

Agile Myth #1: “Agile is a Methodology”

First of all, if you look up the word “methodology” in the dictionary, it says, “study of methods”. When people in the technical or research fields say “methodology”, they really mean “process”. So what is a process? A process is basically a set of instructions:  You follow Step 1, Step 2, Step 3… You use this activity in situation A, you use another activity in situation B. You use this document template, that design notation, etc.  Now process is essential to writing good software. All the “Agile Methodologies” – Scrum, XP, Kanban, etc. – prescribe processes. However, what differentiates Agile processes from other processes like RUP (which is a very good process if evaluated solely on process) is the underlying values and principles. So Scrum, XP, Kanban – these are processes that are founded on Agile. But Agile itself is not a process. Rather, Agile is the values and principles that allow software development processes to be successful. Read the Agile Manifesto, the Principles Behind the Agile Manifesto, the Values of Extreme Programming, the 7 Key Principles of Lean Software Development. Agile processes try to reduce the amount of process in software development, but something has to replace the what has been removed, or the remaining lightweight processes fall apart. It is values and principles that replace the heavyweight processes that have been removed, so that the lightweight processes have a foundation to stand. Let me go through some of the values and principles common to many Agile processes: Customer Value Kent Beck, the inventor of XP, said the reason he invented XP was “to heal the divide between business people and programmers.” Traditional processes tend to put development teams and business stakeholders at odds with each other, which is wrong, since the very purpose of a development team is to create something that is valuable and useful for the business stakeholders. Agile reminds development teams of this primary purpose – that we exist to create things of value to our customers. This is the main motivation for everything we do – why we solicit feedback, why we deliver in short iterations, why we allow changes even late in a project, why we place such strong emphasis on testing… The first characteristic of a team that is really Agile, as opposed to one that is just going through the motions of Agile, is that the members of the team are primarily motivated by the desire to create things of value for their customers. Evidence-Based Decision-Making In Agile, emphasis is placed on making decisions based on evidence. We make estimates after we’ve done a few iterations and have an idea of our team’s velocity, not estimates just pulled out of someone’s butt without any basis. We derive requirements based on actual customer feedback, not what we think our customers will want even if the customers have not had the chance to try out a working system. We base our architectures on Spikes – small prototypes which we subject to analysis and tests – not just vendor documentation or articles that we’ve read, or whatever is the hype technology of the year. Technical Excellence Of the 17 authors of the Agile Manifesto, many of them are thought leaders in the engineering side of software development – Kent Beck is the creator of JUnit and Test-Driven Development, Martin Fowler has written several books on Design Patterns and Refactoring, Robert Martin has written several books on Object-Oriented Design and code quality… A lot of people think Agile is mainly about project management, but the creators of Agile put equal if not greater emphasis on engineering as well. Feedback, Visibility, Courage Agile is about bringing up problems early and often, when they’re still easier to fix. We have stand-up meetings not so that we can update the project manager of our work, but so that we can bring up issues that others can help us with. We have big visible charts so that everyone, not just in the team but in the organization, knows about the progress of the work and if there are any issues. We emphasize testing so that we can discover issues earlier rather than later. For all this to work, there needs to be a culture of courage. This is especially difficult for Filipino culture. When there’s a problem, we Filipinos tend to keep it to ourselves and fix it ourselves – we’ll work unpaid overtime and weekends to try to fix a problem, but when we fail the problem just ends up bigger then when we started. A culture of courage is fundamental in creating a culture and process of feedback and visibility. Eliminating Waste One way we eliminate waste is with requirements. When we specify a lot of requirements upfront, a lot of work in specifying and building the requirements gets wasted, since it’s only when customers are actually able to use a product do we realize that a lot of the requirements were wrong. Instead, in Agile, we specify enough requirements to build the smallest releasable system, so that we can get reliable feedback from customers before building more requirements. Another example is by limiting a team’s “Work in Process” or WIP. For example, if the team notices that number of stories are “in process” exceeds their self-designated “WIP Limit”, they stop and check what’s causing the problem. If, for example, the problem is that the developers are building more features than the testers can test, the developers should stop working and instead help with the testing. Otherwise, untested work just piles up behind the testers as waste. Human Interaction One of the most wasteful things I’ve seen is how people hide behind emails, documents and issue-tracking tools to communicate, and this just leads to issues dragging on and on. A single issue discussed over email might end up being a thread of over a hundred emails, across several departments separated by different floors, with multiple managers CC’d to the discussion. Often, many of these issues that drag on for days or weeks could be resolved in minutes with people just having a conversation with one another. This is why Agile advocates a co-located, interdisciplinary team. A developer can just talk to the DBA across the table in order to resolve an issue within minutes, in less time than it would have taken the issue to be typed up on an email or as a ticket. This is why requirements are initially written as very brief User Stories, to serve simply as a placeholder for face-to-face discussions between the Product Owner and the development team. This is why stand-up meetings are preferred to status reports, because spontaneous collaboration can be initiated there, whereas status reports are largely ignored except for the most meticulous of project managers. Others There’s a number of other principles and values that I haven’t covered here, that are probably much better explained at their respective sources, anyway. Again, I recommend reading up on the Agile Manifesto, the Principles Behind the Agile Manifesto, the Values of Extreme Programming, and the 7 Key Principles of Lean Software Development. This is the first in a series of twelve myths of Agile that I will be discussing, based on my presentation at SofTech 2014 software engineering conference. Hang on for my next blog post, where I will discuss Agile Myth #2: “Agile is About Project Management”.Reference: Agile Myth #1: “Agile is a Methodology” from our JCG partner Calen Legaspi at the Calen Legaspi blog....
software-development-2-logo

Through The Looking Glass Architecture Antipattern

An anti-pattern is a commonly recurring software design pattern that is ineffective or counterproductive. One architectural anti-pattern I’ve seen a number of times is something I’ll call the “Through the Looking Glass” pattern. It’s named so because APIs and internal representations are often reflected and defined by their consumers. The core of this pattern is that the software components are split in a manner that couples multiple components in an inappropriate manner. An example is software that is split between a “front end” and a “back end”, but they both use the same interfaces and/or value objects to do their work. This causes a situation where almost every back end change requires a front-end change and visa versa. As particularly painful examples, think of using GWT (in general) or trying to use SOAP APIs through javascript. There are a number of other ways this pattern can show up… most often it can be a team structure problem. Folks will be split by their technical expertise, but then impede progress because the DBA’s who are required to make database changes are not aware of what the front end application needs so the front end developers end up spend a large amount of time “remote controlling” the database team. It can also be a situation where there is a desire for a set of web service APIs are exposed for front end applications, but because the back-end service calls are only in existence for the front end application, there end up being a number of chicken and egg situations. In almost every case, the solution to this problem is to either #1 add a translation layer, #2 #1 simplify the design, or #3 restructure the work such that it can be isolated on a component by component basis. In the web service example above, it’s probably more appropriate to use direct integration for the front end and expose web services AFTER the front end work has been done. For the database problem, the DBA probably should be embedded with the front-end developer and “help” with screen design so that they have a more complete understanding of what is going on. An alternate solution to the database problem might be to allow the front end developer to build the initial database (if they have the expertise) and allow the DBA to tune the design afterwords. I find that it’s most often easiest and most economical to add a translation layer between layers than trying to unify the design unless the solution space can be clearly limited in scope and scale. I say this because modern rapid development frameworks ([g]rails, play…) now support this in a very simple manner and there is no great reason to NOT do it…except maybe ignorance.Reference: Through The Looking Glass Architecture Antipattern from our JCG partner Mike Mainguy at the mike.mainguy blog....
java-logo

Java’s Volatile Modifier

A while ago I wrote a Java servlet Filter that loads configuration in its init function (based on a parameter from web.xml). The filter’s configuration is cached in a private field. I set the volatile modifier on the field. When I later checked the company Sonar to see if it found any warnings or issues in the code I was a bit surprised to learn that there was a violation on the use of volatile. The explanation read:        Use of the keyword ‘volatile’ is generally used to fine tune a Java application, and therefore, requires a good expertise of the Java Memory Model. Moreover, its range of action is somewhat misknown. Therefore, the volatile keyword should not be used for maintenance purpose and portability.I would agree that volatile is misknown by many Java programmers. For some even unknown. Not only because it’s never used much in the first place, but also because it’s definition changed since Java 1.5. Let me get back to this Sonar violation in a bit and first explain what volatile means in Java 1.5 and up (until Java 1.8 at the time of writing). What is Volatile? While the volatile modifier itself comes from C, it has a completely different meaning in Java. This may not help in growing an understanding of it, googling for volatile could lead to different results. Let’s take a quick side step and see what volatile means in C first. In the C language the compiler ordinarily assumes that variables cannot change value by themselves. While this makes sense as default behavior, sometimes a variable may represent a location that can be changed (like a hardware register). Using a volatile variable instructs the compiler not to apply these optimizations. Back to Java. The meaning of volatile in C would be useless in Java. The JVM uses native libraries to interact with the OS and hardware. Further more, it is simply impossible to point Java variables to specific addresses, so variables actually won’t change value by themselves. However, the value of variables on the JVM can be changed by different threads. By default the compiler assumes that variables won’t change in other threads. Hence it can apply optimizations such as reordering memory operations and caching the variable in a CPU register. Using a volatile variable instructs the compiler not to apply these optimizations. This guarantees that a reading thread always reads the variable from memory (or from a shared cache), never from a local cache. Atomicity Further more on a 32 bit JVM volatile makes writes to a 64 bit variable atomic (like longs and doubles). To write a variable the JVM instructs the CPU to write an operand to a position in memory. When using the 32 bit instruction set, what if the size of a variable is 64 bits? Obviously the variable must be written with two instructions, 32 bits at a time. In multi-threaded scenarios another thread may read the variable half way through the write. At that point only first half of the variable is written. This race-condition is prevented by volatile, effectively making writes to 64 bit variables atomic on 32 bit architectures. Note that above I talked about writes not updates. Using volatile won’t make updates atomic. E.g. ++i when i is volatile would read the value of i from the heap or L3 cache into a local register, inc that register, and write the register back into the shared location of i. In between reading and writing i it might be changed by another thread. Placing a lock around the read and write instructions makes the update atomic. Or better, use non-blocking instructions from the atomic variable classes in the concurrent.atomic package. Side Effect A volatile variable also has a side effect in memory visibility. Not just changes to the volatile variable are visible to other threads, but also any side effects of the code that led up to the change are visible when a thread reads a volatile variable. Or more formally, a volatile variable establishes a happens-before relationship with subsequent reads of that variable. I.e. from the perspective of memory visibility writing a volatile variable effectively is like exiting a synchronized block and reading a volatile variable like entering one. Choosing Volatile Back to my use of volatile to initialize a configuration once and cache it in a private field. Up to now I believe the best way to ensure visibility of this field to all threads is to use volatile. I could have used AtomicReference instead. Since the field is only written once (after construction, hence it cannot be final) atomic variables communicate the wrong intent. I don’t want to make updates atomic, I want to make the cache visible to all threads. And for what it’s worth, the atomic classes use volatile too. Thoughts on this Sonar Rule Now that we’ve seen what volatile means in Java, let’s talk a bit more about this Sonar rule. In my opinion this rule is one of the flaws in configurations of tools like Sonar. Using volatile can be a really good thing to do, if you need shared (mutable) state across threads. Sure thing you must keep this to a minimum. But the consequence of this rule is that people who don’t understand what volatile is follow the recommendation to not use volatile. If they remove the modifier effectively they introduce a race-condition. I do think it’s a good idea to automatically raise red flags when misknown or dangerous language features are used. But maybe this is only a good idea when there are better alternatives to solve the same line of problems. In this case, volatile has no such alternative. Note that in no way this is intended as a rant against Sonar. However I do think that people should select a set of rules that they find important to apply, rather than embracing default configurations. I find the idea to use rules that are enabled by default, a bit naive. There’s an extremely high probability that your project is not the one that tool maintainers had in mind when picking their standard configuration. Furthermore I believe that as you encounter a language feature that you don’t know, you should learn about it. As you learn about it you can decide if there are better alternatives. Java Concurrency in Practice The de facto standard book about concurrency in the JVM is Java Concurrency in Practice by Brain Goetz. It explains the various aspects of concurrency in several levels of detail. If you use any form of concurrency in Java (or impure Scala) make sure you at least read the former three chapters of this brilliant book to get a decent high-level understanding of the matter.Reference: Java’s Volatile Modifier from our JCG partner Bart Bakker at the Software Craft blog....
software-development-2-logo

Test Attribute #2: Readability

This is the 2nd post on test attributes that were described in the now famous “How to test your tests” post. We often forget the most of the value we get from tests come after we’ve written them. Sure, TDD helps design the code, but let’s face it, when everything works for the first time, our tests become the future guardians of our code. Once in place, we can change the code, hopefully for the better, knowing that everything still works. But if (and when) something breaks, there’s work to be done. We need to understand what worked before that doesn’t now. We then need to analyze the situation: According to what we’re doing right now, should we fix this problem? Or is this new functionality that we now need to cover with new tests, throwing away the old one? Finally there’s coding and testing again, depending on the result of our analysis. The more we move further in time, the tests and code get stale in our mind. Then we forget about them completely. The cost of the changes rises. The analysis phase becomes longer, because we need to reacquaint ourselves with the surroundings. We need to re-learn what still works, and what stopped working. Even if we knew, we don’t remember which  changes can cause side effects, and how those will work out. Effective tests minimize this process. They need to be readable. Readability is subjective.  What I find readable now (immediately after I wrote it), will not seem so in 6 months. Let alone to someone else. So instead of trying to define test readability, let’s break it down to elements we care about, and can evaluate. What A Name The most important part of a test (apart from testing the right thing) is its name. The reason is simple: When a test breaks, the name is the first thing we see when the test fails. This is the first clue we get that something is wrong, and therefore, it needs to tell us as much as possible. The name of a test should include (at least) the specific scenario and the expected result of our test. If we’re testing APIs, it should say that too. For example: @Test public void divideCounterBy5Is25() { ... I can understand what the test does (a scenario about dividing Counter), the details (division by 5) and the expected result for this scenario (25). If it sounds like a sentence – even better. Good names come from verbally describing them. It doesn’t matter if you use capitalization, underscore, or whatever you choose. It is important that you use the same convention that your team agrees on. Names should also be specific enough to mentally discern from other sibling tests. So, in our example: @Test public void divideCounterBy0Throws() { ... This test is similar enough to the first name to identify it as a “sibling” scenario, because of the resemblance in the prefix. The specific scenario and result are different. It is important, because when those two will appear together in the test runner, one fails and one doesn’t, it helps us locate the problem before even starting to debug. These are clues to resolve the problem. What A Body If our names don’t help locate the problem, the test body should fill the gaps. It should contain all the information needed to understand the scenarios. Here are a few tips to make test code readable:Tests should be short. About 10-15 lines short. If the setup is long, extract it to functions with descriptive names. Avoid using pre-test functions like JUnit’s @Before or MSTest [TestInitialize]. Instead use methods called directly from the test. When you look at the code, setup and tear down need to be visible, otherwise, you’ll need to search further, and assume even more. Debugging tests with setup methods is no fun either, because you enter the test in a certain context that may surprise you. Avoid using base test classes.  They too hide information relevant for understanding the scenarios. Preferring composition over inheritance works here too. Make the assert part stand out. Make sure that body and name align.Analysis takes time, and the worst kind (and slowest) requires debugging. Tests (and code) should be readable enough to help us bypass this wasteful process. Better Readability In Minutes We are biased and so think that we write code and tests that are so good, everyone can understand it. We’re often wrong. In agile, feedback is the answer. Use the “Law of the Third Ear”. Grab an unsuspecting colleague’s ear, and pull it close to your screen, and she can tell you if she understands what the test does. Even better, pair while writing the tests. Feedback comes us a by product and you get better tests. Whatever you do, don’t leave it for later. And don’t use violence if you don’t need to. Make tests readable now, so you can read them later.Reference: Test Attribute #2: Readability from our JCG partner Gil Zilberfeld at the Geek Out of Water blog....
software-development-2-logo

The Life(Cycles) of UX/UI Development

It recently occurred to me that not one of the dozens and dozens of user interfaces I’ve worked on over the years, had the same methodology/lifecycle.  Many of those were results of the environments under which they were constructed: startup, BIG company, government contract, side-project, open-source, freelance, etc. But the technology also played a part in choosing the methodology we used. Consider the evolution of UI technology for a moment. Back in the early days of unix/x-windows, UI development was a grind.  It wasn’t easy to relocate controls and re-organize interactions.  Because of that, we were forced to first spend some time with a napkin and a sharpie, and run that napkin around to get feedback, etc. The “UX” cycle was born. Then came along things like Visual Basic/C++ and WYSIWIG development. The UI design was literally the application. Drag a button here. Double click. Add some logic. Presto, instant prototype… and application. You could immediately get feedback on the look and feel, etc. It was easy to relocate, reorganize things and react to user feedback. What happened to the “UX” cycle?  It collapsed into the development cycle. The discipline wasn’t lost, it was just done differently, using the same tools/framework used for development. Then, thanks to Mr.Gore, the world-wide web was invented, bringing HTML along with it. I’m talking early here, the days of HTML sans javascript frameworks. (horror to think!)  UI development was thrown back into the stone ages. You needed to make sure you got your site right, because adjusting look and feel often meant “rewrite” the entire web site/application. Many of the “roundtrip” interactions and MVC framework, even though physically separated from the browser,  was entwined in the flow/logic of the UI. (Spring Web Flow anyone? http://projects.spring.io/spring-webflow/) In such a world, again — you wanted to make sure you got it right, because adjustments were costly.   Fortunately, in the meantime the UX discipline and their tools had advanced. It wasn’t just about information display, it was about optimizing the interactions.  The tools were able to not only play with look, but they could focus on and mock out feel. We could do a full UX design cycle before any code was written. Way cool. Once blessed, the design is/was handed off to the development team, and implementation began. Fast forward to present day.  JavaScript frameworks have come a long way.  I now know people that can mockup user experience *in code*, faster and more flexibly than the traditional wire-framing tools. This presents an opportunity to once again collapse the toolset and smooth the design/development process into one cohesive, ongoing process instead of the somewhat disconnected, one-time traditional handoff. I liken this to the shift that took place years ago for application design. We used to sit down and draw out the application architecture and the design before coding: class hierarchies, sequence diagrams, etc. (Rational Rose and UML anyone?). But once the IDE’s advanced enough, it became faster to code the design than to draw it out. The disciplines of architecture and design didn’t go away, they are just done differently now. Likewise with UX.  User experience and design are still of paramount importance.  And that needs to include user research, coordination with marketing, etc.  But can we change the toolset at this point, so the design and development can be unified as one?  If so, imagine the smoothed, accelerated design->development->delivery (repeat) process we could construct! For an innovative company like ours, that depends heavily on time-to-market, that accelerated process is worth a ton. We’ll pay extra to get resources that can bridge that gap between UX and development, and play in both worlds. (Provided we don’t need to sacrifice on either!) On that note, if you think you match that persona. Let me know. We have a spot for you! And if you are a UXer, it might be worth playing around with angular and bootstrap to see how easy things have become. We don’t mind on the job training!Reference: The Life(Cycles) of UX/UI Development from our JCG partner Brian ONeill at the Brian ONeill’s Blog blog....
agile-logo

What we forget about the Scientific Method

I get fed up hearing other Agile evangelists champion The Scientific Method, I don’t disagree with them, I use it myself but I think they are sometimes guilty of overlooking how the scientific method is actually practiced by scientists and researchers. Too often the scientific approach is made to sound simple, it isn’t. First lets define the scientific method. Perhaps rather than call it “scientific method” it is better called “experimentation.” What the Agile Evangelists of my experience are often advocating is running an experiment – perhaps several experiments in parallel but more likely in sequence. The steps are something like this:    Propose a hypothesis, e.g. undertaking monthly software releases instead of bi-monthly will result in a lower percentage of release problems Examine the current position: e.g. find the current figures for release frequency and problems, record these Decide how long you want to run the experiment for, e.g. 6 months Introduce the change and reserve any judgement until the end of the experiment period Examine the results: recalculate the figures and compare these with the original figures Draw a conclusion based on observation and dataI agree with all of this, I think its great, but… Lets leave aside problems of measurement, problems of formulating the hypothesis, problems of making changes and propagation problems (i.e. the time it takes for changes to work though the system). These are not minor problems and they do make me wonder about applying the scientific method in the messy world of software and business but lets leave them to one side for the moment. Lets also leave aside the so-called Hawthorne Effect – the tendency for people to change and improve their behaviour because know they are in an experiment. Although the original Hawthorne experiments were shown to be flawed some time ago the effect might still be real. And the flaws found in the Hawthorne experiments should remind us that there may be other factors at work which we have not considered. Even with all these caveats I’m still prepared to accept an experimental approach to work has value. Sometimes the only way to know whether A or B is the best answer is to actually do A and do B and compare the results. But, this is where my objections start…. There are two important elements missing from the way Agile Evangelists talk about the scientific method. When real scientists – and I include social-scientists here – do an experiment there is more to the scientific method than the experiment and so there should be in work too. #1: Literature review – standing on the shoulders of others Before any experiment is planned scientists start by reviewing what has gone before. They go to the library, sometimes books but journals are probably more up to date and often benefit from stricter peer review. They read what others have found, they read what others have done before, the experiments and theories devised to explain the results. True your business, your environment, your team are all unique and what other teams find might not apply to you. And true you might be able to find flaws in their research and their experiments. But that does not mean you should automatically discount what has gone before. If other teams have found consistent results with an approach then it is possible yours will too. The more examples of something working the more likely it will work for you. Why run an experiment if others have already found the result? Now I’m happy to agree that the state of research on software development is pitiful. Many of those who should be helping the industry here, “Computer Science” and “Software Engineering” departments in Universities don’t produce what the industry needs. (Ian Sommerville’s recent critique on this subject is well worth reading “The (ir)relevance of academic software engineering research”). But there is research out there. Some from University departments and some from industry. Plus there is a lot of research that is relevant but is outside the computing and software departments. For example I have dug up a lot of relevant research in business literature, and specifically on time estimation in psychology journals (see my Notes on Estimation and Retrospective Estimation and More notes on Estimation Research.) As people used to dealing with binary software people might demand a simple “Yes this works” or “No it doesn’t” and those suffering from physics envy may demand rigorous experimental research but little of the research of this type exists in software engineering. Much of software engineering is closer to psychology, you can’t conduct the experiments that would give these answers. You have to use statistics and other techniques and look at probabilities. (Notice I’ve separated computer science from software engineering here. Much of computer science theory (e.g. sort algorithm efficiency, P and NP problems, etc.) can stand up with physics theory but does not address many of the problems practicing software engineers face.) #2: Clean up the lab I’m sure most of my readers did some science at school. Think back to those experiments, particularly the chemistry experiments. Part of the preparation was to check the equipment, clean any that might be contaminated with the remains of the last experiment, ensure the workspace was clear and so on. I’m one of those people who doesn’t (usually) start cooking until they have tidies the kitchen. I need space to chop vegetables, I need to be able to see what I’m doing and I don’t want messy plates getting in the way. There is a word for this: Mise en place, its a French expression which according to Wikipedia means: “is a French phrase which means “putting in place”, as in set up. It is used in professional kitchens to refer to organizing and arranging the ingredients (e.g., cuts of meat, relishes, sauces, par-cooked items, spices, freshly chopped vegetables, and other components) that a cook will require for the menu items that are expected to be prepared during a shift.” (Many thanks to Ed Sykes for telling me a great term.) And when you are done with the chemistry experiment, or cooking, you need to tidy up. Experiments need to include set-up and clean-up time. If you leave the lab a mess after every experiment you will make it more difficult for yourself and others next time. I see the same thing when I visit software companies. There is no point in doing experiments if the work environment is a mess – both physically and metaphorically. And if people leave a mess around when they have finished their work then things will only get harder over time. There are many experiments you simply can’t run until you have done the correct preparation. An awful lot of the initial advice I give to companies is simply about cleaning up the work environment and getting them into a state where they can do experiments. Much of that is informed by reference to past literature and experiments. For example:Putting effective source code control and build systems in place Operating in two week iterations: planning out two weeks of work, reviewing what was done and repeating Putting up a team board and using it as a shared to-do list Creating basic measurement tools, whether they be burn-down charts, cumulative flow diagrams or even more basic measurementsYou get the idea? Simply tidying up the work environment and putting a basic process in place, one based on past experience, one which better matches the way work actually happens can alone bring a lot of benefit to organizations. Some organizations don’t need to get into experiments, they just need to tidy up. And, perhaps unfortunately, that is where is stops for some teams. Simply doing the basics better, simply tidying up, removes a lot of the problems they had. It might be a shame that these teams don’t go further, try more but that might be good enough for them. Imagine a restaurant that is just breaking even, the food is poor, customers sometimes refuse to pay, the service shoddy so tips are small, staff don’t stay long which makes the whole problem worse, a vicious circle develops. In an effort to cut costs managers keep staffing low so food arrives late and cold. Finally one of the customers is poisoned and the local health inspector comes in. The restaurant has to do something. They were staggering on with the old ways until now but a crisis means something must be done. They clean the kitchen, they buy some new equipment, they let the chef buy the ingredients he wants rather than the cheapest, they rewrite the menu to simplify their offering. They don’t have to do much and suddenly the customers are happier: the food is warmer and better, the staff are happier, a virtuous circle replaces a vicious circle. How far the restaurant owners want to push this is up to them. If they want a Michelin star they will go a long way, but if this is the local greasy spoon cafe what is the point? – It is their decision. They don’t need experiments, they only need the opening of the scientific method, the bit that is too often overlooked. Some might call it “Brilliant Basics” but you don’t need to be brilliant, just “Good Basics.” (Remember my In Search of Mediocracy post?). I think the scientific method is sometimes, rightly or wrongly, used as a backdoor in an attempt to introduce change. To lower resistance and get individuals and teams to try something new: “Lets try X for a month and then decide if it works.” Thats can be a legitimate approach. But dressing it up in the language of science feels dishonest. Lets have less talk about “The Scientific Method” and more talk about “Tidying up the Kitchen” – or is it better in French? Mise en place…. come to think of it, don’t the Lean community have Japanese word for this? Pika pika.Reference: What we forget about the Scientific Method from our JCG partner Allan Kelly at the Agile, Lean, Patterns blog....
junit-logo

An alternative approach of writing JUnit tests (the Jasmine way)

Recently I wrote a lot of Jasmine tests for a small personal project. It took me some time until I finally got the feeling of getting the tests right. After this, I always have a hard time when switching back to JUnit tests. For some reason JUnit tests did no longer feel that good and I wondered if it would be possible to write JUnit tests in a way similar to Jasmine. Jasmine is a popular Behavior Driven Development testing framework for JavaScript that is inspired by RSpec (a Ruby BDD testing Framework). A simple Jasmine test looks like this:   describe('AudioPlayer tests', function() {   var player;  beforeEach(function() {     player = new AudioPlayer();   });      it('should not play any track after initialization', function() {     expect(player.isPlaying()).toBeFalsy();   });      ... }); The describe() function call in the first line creates a new test suite using the description AudioPlayer tests. Inside a test suite we can use it() to create tests (called specs in Jasmine). Here, we check if the isPlaying() method of AudioPlayer returns false after creating a new AudioPlayer instance. The same test written in JUnit would look like this: public class AudioPlayerTest {   private AudioPlayer audioPlayer;  @Before    public void before() {     audioPlayer = new AudioPlayer();   }  @Test   void notPlayingAfterInitialization() {     assertFalse(audioPlayer.isPlaying());   }      ... } Personally I find the Jasmine test much more readable compared to the JUnit version. In Jasmine the only noise that does not contribute anything to the test are the braces and the function keyword. Everything else contains some useful information. When reading the JUnit test we can ignore keywords like void, access modifiers (private, public, ..), annotations and irrelevant method names (like the name of the method annotated with @Before). In addition to that, test descriptions encoded in camel case method names are not that great to read. Besides increased readability I really like Jasmine’s ability of nesting test suites. Let’s look at an example that is a bit longer: describe('AudioPlayers tests', function() {   var player;  beforeEach(function() {     player = new AudioPlayer();   });      describe('when a track is played', function() {     var track;        beforeEach(function() {       track = new Track('foo/bar.mp3')       player.play(track);     });          it('is playing a track', function() {       expect(player.isPlaying()).toBeTruthy();     });          it('returns the track that is currently played', function() {       expect(player.getCurrentTrack()).toEqual(track);     });   });      ... }); Here we create a sub test suite that is responsible for testing the behavior when a Track is played by AudioPlayer. The inner beforeEach() call is used to set up a common precondition for all tests inside the sub test suite. In contrast, sharing common preconditions for multiple (but not all) tests in JUnit can become cumbersome sometimes. Of course duplicating the setup code in tests is bad, so we create extra methods for this. To share data between setup and test methods (like the track variable in the example above) we then have to use member variables (with a much larger scope). Additionally we should make sure to group tests with similar preconditions together to avoid the need of reading the whole test class to find all relevant tests for a certain situation. Or we can split things up into multiple smaller classes. But then we might have to share setup code between these classes… If we look at Jasmine tests we see that the structure is defined by calling global functions (like describe(), it(), …) and passing descriptive strings and anonymous functions. With Java 8 we got Lambdas, so we can do the same right? Yes, we can write something like this in Java 8: public class AudioPlayerTest {   private AudioPlayer player;      public AudioPlayerTest() {     describe("AudioPlayer tests", () -> {       beforeEach(() -> {         player = new AudioPlayer();       });      it("should not play any track after initialization", () -> {         expect(player.isPlaying()).toBeFalsy();       });     });   } } If we assume for a moment that describe(), beforeEach(), it() and expect() are statically imported methods that take appropriate parameters, this would at least compile. But how should we run this kind of test? For interest I tried to integrate this with JUnit and it turned out that this actually very easy (I will write about this in the future). The result so far is a small library called Oleaster. A test written with Oleaster looks like this: import static com.mscharhag.oleaster.runner.StaticRunnerSupport.*; ...@RunWith(OleasterRunner.class) public class AudioPlayerTest {   private AudioPlayer player;      {     describe("AudioPlayer tests", () -> {       beforeEach(() -> {         player = new AudioPlayer();       });            it("should not play any track after initialization", () -> {         assertFalse(player.isPlaying());       });     });   } } Only a few things changed compared to the previous example. Here, the test class is annotated with the JUnit @RunWith annotation. This tells JUnit to use Oleaster when running this test class. The static import of StaticRunnerSupport.* gives direct access to static Oleaster methods like describe() or it(). Also note that the constructor was replaced by an instance initializer and the Jasmine like matcher is replaced with by a standard JUnit assertion. There is actually one thing that is not so great compared to original Jasmine tests. It is the fact that in Java a variable needs to be effectively final to use it inside a lambda expression. This means that the following piece of code does not compile: describe("AudioPlayer tests", () -> {   AudioPlayer player;   beforeEach(() -> {     player = new AudioPlayer();   });   ... }); The assignment to player inside the beforeEach() lambda expression will not compile (because player is not effectively final). In Java we have to use instance fields in situations like this (like shown in the example above). In case you worry about reporting: Oleaster is only responsible for collecting test cases and running them. The whole reporting is still done by JUnit. So Oleaster should cause no problems with tools and libraries that make use of JUnit reports. For example the following screenshot shows the result of a failed Oleaster test in IntelliJ IDEA:If you wonder how Oleaster tests look in practice you can have a look at the tests for Oleaster (which are written in Oleaster itself). You can find the GitHub test directory here. Feel free to add any kind of feedback by commenting to this post or by creating a GitHub issue.Reference: An alternative approach of writing JUnit tests (the Jasmine way) from our JCG partner Michael Scharhag at the mscharhag, Programming and Stuff blog....
jsf-logo

How to get JSON response from JSF?

Many JavaScript widgets expect data and options in JSON format. Nowadays, it is really easy to choose a cool widget and wrap it in a composite component. But the first question is how to send an AJAX request and to recieve a response in a proper JSON format. This question is often raised by JSF users. All that you need is a XHTML facelet like this one:             <f:view encoding="UTF-8" contentType="text/html" xmlns="http://www.w3.org/1999/xhtml" xmlns:h="http://xmlns.jcp.org/jsf/html" xmlns:f="http://xmlns.jcp.org/jsf/core"> <h:outputText value="#{stationView.getClosestStations(param.longitude, param.latitude)}" escape="false"/> </f:view> Please consider the contentType=”text/html” (application/json will not work here) and escape=”false” in the h:outputText. The method getClosestStations() in the bean StationView produces an JSON output for a list of special Java objects. I advise to use the Gson library in order to serialize any Java object to JSON. Short example: String[] strings = {"abc", "def", "ghi"}; Gson gson = new Gson(); gson.toJson(strings); ==> prints ["abc", "def", "ghi"] The XHTML file above is located under the web context. Say, under the path /rest/stations.xhtml. Ajax call in your JavaScript code should look like this one: $.ajax({ url: requestContextPath + '/rest/stations.xhtml', type: "GET", data: { "longitude": x, "latitude": y }, dataType: "json", success: function (data) { $.each(data, function (i, station) { ... }); }, error: function () { ... } }); Please refer the jQuery docu for more information regarding $.ajax. Note: if you omit dataType: “json”, you have to parse the JSON string manually. success: function (data) { $.each($.parseJSON(data), function (i, station) { ... }); } The response is a pure JSON string (no HTML tags) like this one: [{"latitude":46.947045,"longitude":7.443922,"distanz":110,"name":"Bern, Bundesplatz"},{....},...] Need more examples for JSON response in JSF? In one of my next posts I will probably explain how to implement a cool autocomplete component without writing too much code.Reference: How to get JSON response from JSF? from our JCG partner Oleg Varaksin at the Thoughts on software development blog....
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books