Featured FREE Whitepapers

What's New Here?


Four laws of robust software systems

Murphy’s Law (“If anything can go wrong, it will”) was born at Edwards Air Force Base in 1949 at North Base. It was named after Capt. Edward A. Murphy, an engineer working on Air Force Project MX981, (a project) designed to see how much sudden deceleration a person can stand in a crash. One day, after finding that a transducer was wired wrong, he cursed the technician responsible and said, “If there is any way to do it wrong, he’ll find it.”For that described reason it may be good to put some quality assurance process in place. I could also call this blog “the four laws of steady software quality”. It’s about some fundamental techniques that can help to achieve superior quality over a longer distance. This is particularly important if you’re developing some central component that will cause serious damage if it fails in production. OK, here is my (never final and not holistic) list of practical quality assurance tips.Law 1: facilitate changeThere is nothing permanent except change. If a system isn’t designed in accordance to this superior important reality, then the probability of failure may increase above average. A widely used technique to facilitate change is the development of a sufficient set of unit tests. Unit testing enables to uncover regressions in existing functionality after changes have been made to a system. It also encourages to really think about the desired functionality and required design of the component under development.Law 2: don’t rush through the functional testing phaseIn economics, the marginal utility of a good is the gain (or loss) from an increase (or decrease) in the consumption of that good. The law of diminishing marginal utility says, that the marginal utility of each (homogenous) unit decreases as the supply of units increases (and vice versa). The first functional test cases often walk through the main scenarios covering the main paths of the considered software. All the code tested wasn’t executed before. These test cases have a very high marginal utility. Subsequent test cases may walk through the same code ranges except specific sidepaths at specific validation conditions for instance. These test cases may cover three or four additional lines of code in your application. As a result, they will have a smaller marginal utility then the first test cases.My law about functional testing suggests: as long the execution of the next test case yields a significant utility the following applies: the more time you invest into testing the better the outcome! So don’t rush through a functional testing phase and miss out some useful test case (this assumes the special case in which usefulness can be quantified). Try to find the useful test cases that promise a significant gain in perceptible quality. On the other hand, if you’re executing test cases with a negative marginal utility you’re actually investing more effort then you gain in terms of perceptible quality. There is this special (but not uncommon) situation where the client does not run functional tests on systematic bases. This law will then suggest: the longer the application is in the test environment, the better the outcome.Law 3: run (non-functional) benchmark testsAnother peace of good permanent software quality is a regular load test. To make results usable load tests need a defined steady environment and a baseline of measured values (a benchmark). These values are at least: CPU, response time, memory footprint. Load tests of new releases can be compared to those load tests of older releases. That way we can also bypass the often stated requirement that the load test environment needs to have the same capacity parameters then the production environment. In many cases it is possible to see the real big issues with a relatively small set of parallel users (e.g. 50 users).It makes limited sense to do load testing if single user profiling results are bad. Therefore it’s a good idea to perform repeatable profiling test cases with every release. This way profiling results can be compared to each other (again: the benchmark idea). We do CPU and elapsed time profiling as well as memory profiling. Profiling is an activity that runs in parallel to actual development. It makes sence to focus on the main scenarios used regularly in production.Law 4: avoid dependency lock-inThe difference between trouble and severe crisis is the time it takes to fix the problem that causes the trouble. For this reason you may always need a way back to your previous release, you need a fallback scenario to avoid a production crisis with severe business impact. You enable rollback by avoiding dependency lock-in. Runtime-dependencies of your application may exist to neighbouring systems by joint interface or contract changes during development. If you implemented requirements that resulted in changed interfaces and contracts, then you cannot simply roll back, that’s obvious. Therefore you need to avoid too many interface and contract changes. Small release cycles help to reduce dependencies between application versions in one release ’cause less changes are rolled to production. Another counteraction against dependency lock-in is to let neighbouring systems be downwoards compatible for one version.That’s it in terms of robust systems.Reference: “5′ on IT-Architecture: four laws of robust software systems” from our JCG partner Niklas....

OpenShift Express Web Management Console: Getting started

This week the newest release of OpenShift brought two really great features to an already awesome PaaS Cloud provider. First, JBoss AS has been upgraded from 7.0 to 7.1 and the all new Express Web Management Console has been released as a preview. In this article we examine how to use this new console and will help you create and then destroy an application. OverviewFigure 1: follow link to launch the Express ConsoleIn this section we assume you have already registered as an OpenShift user and are logged into the OpenShift Express start page. In figure 1 the Express landing page is shown and if you follow the Express Console link you will be brought to a page that currently shows the old administration console and includes a link to P review the new OpenShift Management Console. Follow this link to the preview as shown for my user in figure 2.Figure 2: preview Express management consoleIt provides an overview of the users existing application, with a Details button for each application. My user has two application already created, one a jBPM web editor project based on JBoss and a second PHP twitter project that makes use of mongodb as a backend, see figure 2. At the top of the applications list, you have a button to Create a New Application. We will be using this button to create an existing project called kitchensinkhtml5, a mobile application from the JBoss project Aerogear. The nice thing about this demo project is that you can view it both in your desktop browsers and in your mobile devices.Figure 3: choose a type of applicationCreate application   Since this user already has created a domain and has existing applications setup, we just need to start by using the Create a New Application button. This takes us to the first of three steps where we will c hoose a type of application, which will be the JBoss Application Server 7.1 chosen by the Select button shown in figure 3.Figure 4: create applicationThe next step is to configure and deploy the application, done by filling in an application name in the provided text box and clicking on the Create Application button. We will be calling this application kitchensinkhtml5, so we fill in that name in the text box and submit to create our new application as shown in figure 4.Figure 5: next stepsOnce we submit our creation request, the OpenShift Express magic is started to setup our new instance with JBoss AS 7.1 started. We are presented with a final screen that is labeled Next Steps which provides information on accessing your application, making code changes, how to manage your application and how to start adding capabilities. As shown in figure 5, we will be pulling in a git clone of our Express application repository so that we can setup our kitchensink application code. As stated in the section making code changes we will clone the repository locally from a shell command line: git clone ssh://8df3de8e983c4b058db372e51bfe5254@kitchensinkhtml5-inthe.rhcloud.com/~/git/kitchensinkhtml5.git/ cd kitchensinkhtml5/ Once that is done we need to pull in our existing kitchensink code base: cd kitchensinkhtml5 git remote add upstream -m master git://github.com/eschabell/kitchensink-html5-mobile-example.git git pull -s recursive -X theirs upstream masterFinally, we push this back upstream to our Express instance as follows: git push We can now view the application at the URL assigned to our Express instance: http://kitchensinkhtml5-{$domainname}.rhcloud.comYou should see the mobile member registration application as shown here in figure 6.Figure 6: mobile applicationDestroy application   A final action that you can do with the new OpenShift Express Web Management Console is to destroy your application. As we only get five instances at a time, you will soon find yourself creating and destroying Express instances with ease.Figure 7: delete applicationAfter logging in as described above and starting the preview of the web management console, you will see your list of existing applications. By selecting an applications Details button you will be shown an overview of the application, see figure 7 for our example editor application we will be destroying.Figure 8: application deletedYou will notice a Delete button in the right top corner of the application overview screen, see figure 7. When selected, you will be asked to confirm that you really want to destroy this application. If you confirm this decision by clicking on the Delete button, your application and Express instance will be cleaned up. You will be returned to the application overview screen, see figure 8, and are ready for your next interaction with the Express Web Administration Console. Summary   In this article we have covered the very basics of the newly released OpenShift Express Web Administration Console. We have shown you how to view your applications, create a new application and how to free up an Express instance by destroying one of your applications. Reference: Getting started with the OpenShift Express Web Management Console from our JCG partner Eric D. Schabell at the Thoughts on Middleware, Linux, software, cycling and other news… blog....

Google Protocol Buffers in Java

Overview Protocol buffers is an open source encoding mechanism for structured data. Developed at Google, it was designed to be language/platform neutral and extensible. In this post, my aim is to cover the basic use of protocol buffers in the context of the Java platform. Protobuffs are faster and simpler than XML and more compact than JSON. Currently, there is support for C++, Java, and Python. However, there are other platforms supported (not by Google) as open source projects –I tried a PHP implementation but it wasn’t fully developed so I stopped using it; nonetheless, support is catching on. With Google announcing support for PHP in Google App Engine, I believe they will take this to next level. Basically, you define how you want your data to be structured once using a .proto specification file. This is analogous to an IDL file or a specification language to describe a software component. This file is consumed by the protocol buffer compiler (protoc) which will generate supporting methods so that you can write and read objects to and from a variety of streams. The message format is very straightforward. Each message type has one or more uniquely numbered fields (we’ll see why this is later). Nested message types have their own set of uniquely numbered fields. Value types can be numbers, booleans, strings, bytes, collections and enumerations (inspired in the Java enum). Also, you can nest other message types, allowing you to structure your data hierarchically in much the same way JSON allows you to. Fields can be specified as optional, required, or repeated. Don’t let the type of the field (e.g enum, int32, float, string, etc) confuse you when implementing protocol buffers in Python. The types in the field are just hints to protoc about how to serialize a fields value and produce the message encoded format of your message (more on this later). The encoded format looks a flatten and compressed representation of your object. You would write this specification the exact same way whether you are using protocol buffers in Python, Java, or C++. Protobuffs are extensible, you can update the structure of your objects at a later time without breaking programs that used the old format. If you wanted to send data over the network, you would encode the data using Protocol Buffer API and then serialize the resulting string. This notion of extensibility is a rather important one since Java, and many other serialization mechanisms for that matter, could potentially have issues with interoperability and backwards compatibility. With this approach, you don’t have to worry about maintaining a serialVersionId field in your code that represents the structure of an object. Maintaining this field is essential as Java’s serialization mechanism will use it as a quick checksum when deserializing objects. As a result, once you have serialized your objects into some file system, or perhaps a blob store, it is risky to make drastic changes to your object structure at a later time. Protocol buffer suffers less from this. So long as you only add optional fields to your objects, you will be able to deserialize old types at which point you will probably upgrade them. Furthermore, you can define a package name for your .proto files with the java_package keyword. This is nice to avoid name collisions from the generated code. Another alternative is to specifically name the generated class file as I did in my example below. I prefixed my generated classes with “Proto” to indicate this was a generated class. Here’s a simple message specification describing a User with an embedded Address message User.proto: option java_outer_classname="ProtoUser";message User {required int32 id = 1; // DB record ID required string name = 2; required string firstname = 3; required string lastname = 4; required string ssn= 5;// Embedded Address message specmessage Address { required int32 id = 1; required string country = 2 [default = "US"];; optional string state = 3; optional string city = 4; optional string street = 5; optional string zip = 6;enum Type { HOME = 0;WORK = 1;}optional Type addrType = 7 [default = HOME];} repeated Address addr = 16; }Let’s talk a bit about the tag numbers you see to the right of each property since they are very important. These tags identify the field order of your message in the binary representation on an object of this specification. Tag values 1 – 15 will be stored as 1 byte, whereas fields tagged with values 16 – 2047 take 2 bytes to encode — not quiet sure why they do this. Google recommends you use tags 1 – 15 for very frequently occurring data and also reserve some tag values in this range for any future updates. Note: You cannot use numbers 19000 though 19999. There are reserved for protobuff implementation. Also, you can define fields to be required, repeated, and optional.From the Google documentation:required: a well-formed message must have exactly one of this field, i.e trying to build a message with a required field uninitialized will throw a RuntimeException. optional: a well-formed message can have zero or one of this field (but not more than one). repeated: this field can be repeated any number of times (including zero) in a well-formed message. The order of the repeated values will be preserved.The documentation warns developers to be cautious about using required, as this types of fields will cause problems if you ever decide to deprecate one. This is a classical backwards compatibility problem that all serialization mechanisms suffer from. Google engineers even recommend using optional for everything. Furthermore, I specified a nested message specification Address. I could have just as easily place this definition outside the User object in the same proto file. So for related message definitions it makes sense to have them all in the same .proto file. Even though the Address message type is not a very good example of this, I would go with a nested type if a message type does not make sense to exist outside of its ‘parent’ object. For instance, if you wanted to serialize a Node of a LinkedList. Then node would in this case be an embedded message definition. It’s up to you and your design. Optional message properties take on default values when they are left out. In particular a type-specific default value is used instead: for strings, the default value is the empty string; for bools, the default value is false; for numeric types, the default value is zero; for enums, the default value is the first value listed in the enum’s type definition (this is pretty cool but not so obvious). Enumerations are pretty nice. They work cross-platform in much the same way as enum works in Java. The value of the enum field can just be a single value. You can declare enumerations inside the message definition or outside as if it was it’s own independent entity. If specified inside a message type, you can expose it another message type via [Message-name].[enum-name]. Protoc When running the protocol buffer compiler against a .proto file, the compiler will generate code for chosen language. It will convert your message types into augmented classes providing, among other things, getters and setters for your properties. The compiler also generates convenience methods to serialize messages to and from output streams and strings. In the case of an enum type, the generated code will have a corresponding enum for Java or C++, or a special EnumDescriptor class for Python that’s used to create a set of symbolic constants with integer values in the runtime-generated class. For Java, the compiler will generate .java files with a fluent design Builder classes for each message type to streamline object creation and initialization. The message classes generated by the compiler are immutable; once built, they cannot be changed. You can read about other platforms (Python, C++) in the resources section with details into field encodings here: https://developers.google.com/protocol-buffers/docs/reference/overview. For our example, we will invoke protoc with the –java_out command line flag. This flag indicates to the compiler the output directory for the generated Java classes –one Java class for each proto file. API The generated API provides support for the following convenience methods:isInitialized() toString() mergeFrom(…) clear()For parsing and serialization:byte[] toByteArray() parseFrom() writeTo(OutputStream) Used in sample code to encode parseFrom(InputStream) Used in sample code to decodeSample Code Let’s set up a simple project. I like to follow the Maven default archetype: protobuff-example/src/main/java/ [Application Code] protobuff-example/src/main/java/gen [Generated Proto Classes] protobuff-example/src/main/proto [Proto file definitions] To generate the protocol buffer classes, I will execute the following command: # protoc --proto_path=/home/user/workspace/eclipse/trunk/protobuff/ --java_out=/home/user/workspace/eclipse/trunk/protobuff/src/main/java /home/user/workspace/eclipse/trunk/protobuff/src/main/proto/User.proto I will show some pieces of the generated code and speak about them briefly. The generated class is quiet large but it’s straightforward to understand. It will provide builders to create instances of User and Address. public final class ProtoUser {public interface UserOrBuilder extends com.google.protobuf.MessageOrBuilder...public interface AddressOrBuilder extends com.google.protobuf.MessageOrBuilder {....}The generated class contains Builder interfaces that makes for really fluent object creation. These builder interfaces have getters and setters for each property specified in our proto file, such as: public String getCountry() { java.lang.Object ref = country_; if (ref instanceof String) { return (String) ref; } else { com.google.protobuf.ByteString bs = (com.google.protobuf.ByteString) ref; String s = bs.toStringUtf8(); if (com.google.protobuf.Internal.isValidUtf8(bs)) { country_ = s; } return s; } }Since this is a custom encoding mechanism, logically all of the fields have custom byte wrappers. Our simple String field, when stored, is compacted using a ByteString which then gets de-serialized into a UTF-8 string. // required int32 id = 1; public static final int ID_FIELD_NUMBER = 1; private int id_; public boolean hasId() { return ((bitField0_ & 0x00000001) == 0x00000001); }In this call we see the importance of the tag numbers we spoke of at the beginning. Those tag numbers seem to represent some sort of bit position that define where the data is located in the byte string. Next we see snippets of the write and read methods I mentioned earlier. Writing an instance to the an output stream: public void writeTo(com.google.protobuf.CodedOutputStream output) throws java.io.IOException {getSerializedSize();if (((bitField0_ & 0x00000001) == 0x00000001)) { output.writeInt32(1, id_); } if (((bitField0_ & 0x00000002) == 0x00000002)) { output.writeBytes(2, getCountryBytes()); .... }Reading from an input stream: public static ProtoUser.User parseFrom(java.io.InputStream input) throws java.io.IOException { return newBuilder().mergeFrom(input).buildParsed(); }This class is about 2000 lines of code. There are other details such as how Enum types are mapped and how repeated types are stored. Hopefully, the snippets that I provided give you a high level idea of the structure of this class. Let’s take a look at some application level code for using the generated class. To persist the data, we can simply do: // Create instance of Address Address addr = ProtoUser.User.Address.newBuilder() .setAddrType(Address.Type.HOME) .setCity("Weston") .setCountry("USA") .setId(1) .setState("FL") .setStreet("123 Lakeshore") .setZip("90210") .build(); // Serialize instance of User User user = ProtoUser.User.newBuilder() .setId(1) .setFirstname("Luis") .setLastname("Atencio") .setName("luisat") .setSsn("555-555-5555") .addAddr(addr) .build(); // Write file FileOutputStream output = new FileOutputStream("target/user.ser"); user.writeTo(output); output.close();Once persisted, we can read as such: User user = User.parseFrom( new FileInputStream("target/user.ser"); System.out.println(user);To run the sample code, use: java -cp .:../lib/protobuf-java-2.4.1.jar app.Serialize ../target/user.ser Protobuff vs XML Google claims that protocol buffers are 20 to 100 times faster (in nanoseconds) than XML and 3 to 10 smaller removing whitespace. However, until there is support and adoption in all platforms (not just the aforementioned 3), XML will be continue to be a very popular serialization mechanism. In addition, not everyone has the performance requirements and expectations that Google users have. An alternative to XML is JSON. Protobuff vs JSON I did some comparison testing to evaluate using Protocol buffers over JSON. The results were quiet dramatic, a simple test reveals that protobuffs are 50%+ more efficient in terms of storage. I created a simple POJO version of my User-Address classes and used the GSON library to encode an instance with the same state as the example above (I will omit implementation details, please check gson project referenced below). Encoding the same user data, I got: -rw-rw-r-- 1 luisat luisat 206 May 30 09:47 json-user.ser -rw-rw-r-- 1 luisat luisat 85 May 30 09:42 user.serWhich is remarkable. I also found this in another blog (see resources below):It’s definitely worth a read. Conclusion and Further Remarks Protocol buffers can be a good solution to cross platform data encoding. With clients written in Java, Python, C++ and many others, storing/sending compressed data is really straightforward. One tricky point to make is: “Remember REQUIRED is forever.” If you go crazy and make every single field of your .proto file required, then it will extremely difficult to delete or edit those fields.Also a bit of incentive, protobuffs are used across Google’s data stores: there are 48,162 different message types defined in the Google code tree across 12,183 .proto files. Protocol Buffers promote good Object Oriented Design, since .proto files are basically dumb data holders (like structs in C++). According to Google documentation, if you want to add richer behavior to a generated class or you don’t have control over the design of the .proto file, the best way to do this is to wrap the generated protocol buffer class in an application-specific class. Finally, remember you should never add behaviour to the generated classes by inheriting from them . This will break internal mechanisms and is not good object-oriented practice anyway. A lot of the information presented here comes from personal experience, other resources, and most importantly google developer code. Please check out the documentation in the resources section. Resourceshttps://developers.google.com/protocol-buffers/docs/overview https://developers.google.com/protocol-buffers/docs/proto https://developers.google.com/protocol-buffers/docs/reference/java-generated https://developers.google.com/protocol-buffers/docs/reference/overview http://code.google.com/p/google-gson/ http://afrozahmad.hubpages.com/hub/protocolbuffersReference: Java Protocol Buffers from our JCG partner Luis Atencio at the Reflective Thought blog....

Java concurrency – Feedback from tasks

Picking up from where I left off in my last post about the java.util.concurrent package, it’s interesting and sometimes mandatory to get feedback from concurrent tasks after they are started. For example imagine an application that has to send email batches, besides from using a multi-threaded mechanism, you want to know how many of the intended emails were successfully dispatched, and during the actual sending process, the real-time progress of the whole batch. To implement this kind of multi-threading with feedback we can use the Callable interface. This interface works mostly the same way as Runnable, but the execution method (call()) returns a value that should reflect the outcome of the performed computation. Let’s first define the class that will perform the actual task: package com.ricardozuasti;import java.util.concurrent.Callable;public class FictionalEmailSender implements Callable<Boolean> { public FictionalEmailSender (String to, String subject, String body){ this.to = to; this.subject = subject; this.body = body; }@Override public Boolean call() throws InterruptedException { // Simulate that sending the email takes between 0 and 0.5 seconds Thread.sleep(Math.round(Math.random()* 0.5 * 1000));// Lets say we have an 80% chance of successfully sending our email if (Math.random()>0.2){ return true; } else { return false; } }private String to; private String subject; private String body; } Notice that your Callable can use any return type, so your task can return whatever info you need. Now we can use a thread pool ExecutorService to send our emails, and since our task is implemented as a Callable, we get a Future reference for each new task we submit for execution. Note that we will create our ExecutorService using a direct constructor instead of a utility method from Executors, this is because using the specific class (ThreadPoolExecutor) provides some methods that will come in handy (not present present in the ExecutorService interface). package com.ricardozuasti;import java.util.ArrayList; import java.util.List; import java.util.concurrent.Future; import java.util.concurrent.LinkedBlockingQueue; import java.util.concurrent.ThreadPoolExecutor; import java.util.concurrent.TimeUnit;public class Concurrency2 {public static void main(String[] args) { try { ThreadPoolExecutor executor = new ThreadPoolExecutor(30, 30, 1, TimeUnit.SECONDS, new LinkedBlockingQueue());List<Future<Boolean>> futures = new ArrayList<Future<Boolean>>(9000);// Lets spam every 4 digit numeric user on that silly domain for (int i = 1000; i < 10000; i++) { futures.add(executor.submit(new FictionalEmailSender(i + '@wesellnumericusers.com', 'Knock, knock, Neo', 'The Matrix has you...'))); }// All tasks have been submitted, wen can begin the shutdown of our executor System.out.println('Starting shutdown...'); executor.shutdown();// Every second we print our progress while (!executor.isTerminated()) { executor.awaitTermination(1, TimeUnit.SECONDS); int progress = Math.round((executor.getCompletedTaskCount() * 100) / executor.getTaskCount());System.out.println(progress + '% done (' + executor.getCompletedTaskCount() + ' emails have been sent).'); }// Now that we are finished sending all the emails, we can review the futures // and see how many were successfully sent int errorCount = 0; int successCount = 0; for (Future<Boolean> future : futures) { if (future.get()) { successCount++; } else { errorCount++; } }System.out.println(successCount + ' emails were successfully sent, but ' + errorCount + ' failed.');} catch (Exception ex) { ex.printStackTrace(); } } } After all tasks are submitted to the ExecutorService, we begin it’s shutdown (preventing new tasks from being submitted) and use a loop (in a real-life scenario you should continue doing something else if possible) to wait until all tasks are finished, calculating and printing the progress made so far on each iteration. Note that you could store the executor reference and query it from other threads any time to calculate and report the process progress. Finally, using the collection of Future references we got for each Callable submitted to the ExecutorService, we can inform the number of emails successfully sent and the number that failed to. This infrastructure is not only easy to use but also promotes clear separation of concerns, providing a pre-defined communication mechanism between the dispatcher program and the actual tasks. Reference: Java concurrency examples – Getting feedback from concurrent tasks from our JCG partner Ricardo Zuasti at the Ricardo Zuasti’s blog blog....

JRockit – JRCMD useful commands

I have been using JRockit since many years 2007. I find it slower than Hotspot, but it has been always better on diagnosis and analysis of problems. Since last summer, I have been working for an international telecommunication systems vendor. We design and implement various products for telecom operators, on top of HP OpenCall Convergent Communication Platform. I am fan of Open Source and Free Software, but this platform is running “only” with JRockit VM. We have tuning it for low latencies and runs very cool, but we have faced various problems, where JRockit has helped us a lot on troubleshooting. I will describe here a few useful commands of JRCMD. JRCMD is a small command-line tool that can be used to interact with a running JRockit instance. Summary:Get a thread dump Memory utilization Memory analysis of heap on a per-class basis State of VM Create a flight recording Produce a heap dumpDetails: 1) Get a thread dump $> jrcmd <pid> print_threads [nativestack=true] This is the usual SIGQUIT handler, which prints all thread stacks. You can also get a thread using the classic way: “$> kill -3 <pid>“ In any case you will get a thread dump like this: ===== FULL THREAD DUMP =============== Thu Jun 21 11:38:19 2012 Oracle JRockit(R) R28.1.4-7-144370-1.6.0_26-20110617-2130-linux-ia32 'http-' id=46791 idx=0x4 tid=17680 prio=5 alive, waiting, native_blocked, daemon -- Waiting for notification on: org/apache/tomcat/util/net/JIoEndpoint$Worker@0x5eef4588[fat lock] at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method) at java/lang/Object.wait(J)V(Native Method) at java/lang/Object.wait(Object.java:485) at org/apache/tomcat/util/net/JIoEndpoint$Worker.await(JIoEndpoint.java:415) ^-- Lock released while waiting: org/apache/tomcat/util/net/JIoEndpoint$Worker@0x5eef4588[fat lock] at org/apache/tomcat/util/net/JIoEndpoint$Worker.run(JIoEndpoint.java:441) at java/lang/Thread.run(Thread.java:662)[optimized] at jrockit/vm/RNI.c2java(IIIII)V(Native Method) -- end of trace '(Signal Handler)' id=2 idx=0x8 tid=21213 prio=5 alive, native_blocked, daemon '(OC Main Thread)' id=3 idx=0xc tid=21214 prio=5 alive, native_waiting, daemon '(GC Worker Thread 1)' id=? idx=0x10 tid=21215 prio=5 alive, daemon '(GC Worker Thread 2)' id=? idx=0x14 tid=21216 prio=5 alive, daemon '(Code Generation Thread 1)' id=4 idx=0x18 tid=21217 prio=5 alive, native_waiting, daemon '(Code Optimization Thread 1)' id=5 idx=0x1c tid=21218 prio=10 alive, native_waiting, daemon '(Code Optimization Thread 2)' id=6 idx=0x20 tid=21219 prio=10 alive, native_waiting, daemon '(VM Periodic Task)' id=7 idx=0x24 tid=21220 prio=10 alive, native_blocked, daemon 'Finalizer' id=8 idx=0x28 tid=21221 prio=8 alive, native_waiting, daemon at jrockit/memory/Finalizer.waitForFinalizees(J[Ljava/lang/Object;)I(Native Method) at jrockit/memory/Finalizer.access$700(Finalizer.java:12)[optimized] at jrockit/memory/Finalizer$4.run(Finalizer.java:189) at java/lang/Thread.run(Thread.java:662) at jrockit/vm/RNI.c2java(IIIII)V(Native Method) -- end of trace 'Reference Handler' id=9 idx=0x2c tid=21222 prio=10 alive, native_waiting, daemon at java/lang/ref/Reference.waitForActivatedQueue(J)Ljava/lang/ref/Reference;(Native Method) at java/lang/ref/Reference.access$100(Reference.java:11) at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:82) at jrockit/vm/RNI.c2java(IIIII)V(Native Method) -- end of trace ... ... 2) Memory utilization $> jrcmd <pid> print_memusage This command can help you with out of memory errors. It analyses (in summary) the memory allocated by java process (including native code). Columns: a) The name of a memory space, b) How much memory is mapped for that space, c) Extra details. A sample run is the following: Total mapped 2110560KB (reserved=6044KB) - Java heap 1572864KB (reserved=0KB) - GC tables 52620KB - Thread stacks 55060KB (#threads=306) - Compiled code 23872KB (used=21690KB) - Internal 776KB - OS 23368KB - Other 257328KB - Java class data 123648KB (malloced=123375KB #169700 in 29996 classes) - Native memory tracking 1024KB (malloced=296KB #8) 3) Memory analysis of heap on a per-class basis $> jrcmd <pid> print_object_summary Prints details of all the instances on the heap on a per-class basis, together with a differential value of how the memory usage has changed. Columns: a) Percentage of heap that objects of this class occupy, b) The total size occupied by the instances of a specific class, c) The number of instances of a specific class, d) The change in size from the first invocation of this command, e) The full name of the class. Check the next sample run, where we might have a problem with objects of class “org/adrianos/MyDTO”: --------- Detailed Heap Statistics: --------- 61.1% 939735k 6772960 +939735k [C 16.4% 252243k 10762404 +252243k java/lang/String 7.0% 107516k 3947228 +107516k [Ljava/lang/String; 4.5% 69265k 369180 +69265k [Ljava/lang/Object; 1.6% 24127k 205889 +24127k org/adrianos/MyDTO 1.3% 19486k 1247140 +19486k java/lang/Long 1.0% 15551k 26621 +15551k [B 0.6% 8871k 9700 +8871k [I 0.6% 8710k 103896 +8710k [Ljava/util/HashMap$Entry; 1537175kB total ------------ End of Detailed Heap Statistics --- 4) State of VM $> jrcmd <pid> print_vm_state This command produce an output similar to the dump file that is normally created when a JRockit instance crashes. It shows various information of VM like command line arguments of java process, uptime, CPU type, state of heap, loaded modules, libc release, etc.. Check the next excerpt of a sample run: Uptime : 5 days, 16:29:55 on Thu Jun 21 12:02:34 2012 Version : Oracle JRockit(R) R28.1.4-7-144370-1.6.0_26-20110617-2130-linux-ia32 CPU : Intel Westmere (HT) SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 Core Intel64 Number CPUs : 8 Tot Phys Mem : 12632571904 (12047 MB) OS version : Red Hat Enterprise Linux Server release 5.5 (Tikanga) Linux version 2.6.18-194.26.1.el5PAE (mockbuild@x86-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Fri Oct 29 14:28:58 EDT 2010 (i686) Thread System: Linux NPTL LibC release : 2.5-stable Java locking : Lazy unlocking enabled (class banning) (transfer banning) State : JVM is running (Main thread has finished) Command Line : -Dprogram.name=run.sh -Xms1536M -Xmx1536M -Djava.net.preferIPv4Stack=true -Xverbose:gc,memory -XverboseLog:/tmp/gc-jrockit.log -Xbootclasspath/p: -XXaggressive -XXcompaction:heapParts=1536 -Xgc:genconcon -Xns:150M -XXgcThreads:2 -XXgcTrigger:50 -XXcompaction:internalPercentage=1.0 -XXcompaction:externalPercentage=1.0 -Xmanagement -Djrockit.managementserver.port=4646 -Dcom.sun.management.jmxremote.ssl=false -Djava.net.preferIPv4Stack=true -Djava.endorsed.dirs=/var/lib/OC/imsc/lib/endorsed -Dsun.java.launcher=SUN_STANDARD com.adrianos.Main Repository : /tmp/2012_06_15_19_32_40_21109 java.home : /usr/java/jrockit-jdk1.6.0_26-R28.1.4-4.0.1.orig/jre StackOverFlow: 0 StackOverFlowErrors have occured OutOfMemory : 0 OutOfMemoryErrors have occured C Heap : Good; no memory allocations have failed GC Strategy : Mode: pausetime, with strategy: genconcon (basic strategy: genconcon) GC Status : OC is not running. Last finished OC was OC#5287. : YC is not running. Last finished YC was YC#16925. YC Promotion : Last YC successfully promoted all objects YC History : Ran 3 YCs before OC#5283. : Ran 3 YCs before OC#5284. : Ran 3 YCs before OC#5285. : Ran 3 YCs before OC#5286. : Ran 3 YCs before OC#5287. : Ran 2 YCs since last OC. Heap : 0x5661a000 - 0xb661a000 (Size: 1536 MB) Compaction : (no compaction area) Allocation : TLA-min: 2048, TLA-preferred: 20480 TLA-waste limit: 2048 NurseryList : 0x869370d8 - 0x941e0390 KeepArea : 0x8da6c078 - 0x941e0390 KA Markers : [ 0x8b4857c8, 0x8da6c078 , 0x941e0390 ] Forbidden A : (none) Previous KA : 0x8b4857c8 - 0x8da6c078 Previous FA : (none) CompRefs : References are 32-bit. ... ... Loaded modules: 08048000-08057193 /usr/java/jrockit-jdk1.6.0_26-R28.1.4-4.0.1.orig/bin/java b7f12000-b7f1262b /usr/java/jrockit-jdk1.6.0_26-R28.1.4-4.0.1.orig/bin/java ... ... 5) Create a flight recording $> jrcmd <pid> start_flightrecording name=myrecord1 filename=/var/tmp/myrecord1.jfr duration=60s compress=true settings=/my/path/xxx.jfs Starts a JRockit Flight Recorder recording that can help you analyse the behaviour of your code and find potentials problems (like bottlenecks). This is really useful to understand what your threads are doing. There re many templates located in the JROCKIT_HOME/jre/lib/jfr directory. 6) Produce a heap dump $> jrcmd <pid> hprofdump filename=/tmp/jrockit1.hprof Produce heap dumps in the popular HPROF format that can be used to resolve memory leaks or have a better understanding of your code. You can analyse this file using the excellent Eclipse Memory Analyzer Tool (MAT) or default Java memory analyser VisualVM. General Tips:You have to be careful with the last 2 commands as they are very useful, but need extra resources from JVM. Avoid to execute them on high traffic hours, except you really need them. Keep in mind that if JVM is at very “difficult” state, it will not permit such actions. MAT is a great tool and I love it. BUT if you want to be fully prepared you have to also play with VisualVM as exists in default installation of Hotspot. This means that VisualVM is always there and you do not need an extra graphical tool to check something simple. I have been in a such case, where I did not have internet access and my laptop use was prohibited.I hope you will find these details useful. Reference: JRockit – JRCMD useful commands from our JCG partner Adrianos Dadis at the Java, Integration and the virtues of source blog....

Sooner or Later: Deliver Early or Minimize Waste

There’s an obvious but important tension in Lean/Agile development around when to make decisions. Between the fundamental Agile position that we should do the most important and most risky work first, and the Lean argument that we should make decisions at the last possible moment. We need to decide early and try things out, iterate to minimize risk and time to market; but we should put off decisions and avoid iterating to keep costs down. Later is better than sooner Lean Development says that in order to avoid waste, we should wait to make decisions as late as possible, especially when we think the stakes may be high. The later that you can make a decision, the more information that you will have on hand to inform your decision, and the more options that you will have to choose from. This means that you should be able to make better quality decisions. Deferred decision making is built into incremental/iterative planning and management. You don’t try to anticipate everything up front. You don’t try to work out all of the details and dependencies and resourcing at the beginning. Instead, you sketch out an outline of a plan with all of the critical dependencies and milestones that you know about, then focus on what’s in front of you as you move ahead. Detailed decisions about scheduling and implementation and re-prioritization are made when you are ready to do the work, just-in-time. Deciding later costs less In Agile Estimating and Planning,Mike Cohn says that “The best way to reduce the cost of change is to implement a feature as late as possible – effectively when there is no more time for change”. Because you are putting off work until later, there’s less chance that new information will come up and force you to change something that you’ve already decided on. Procrastination pays – or, at least, it saves some time and money. You don’t want to put a lot of work into something until you know as much as you can about what you need to do, how you need to do it, or even if you really do need to do it. Build a prototype or run a technical spike first, build a first-of and maybe a second-of something and get the customer using it, get feedback and make sure you have a good idea of what you are doing before you do more typing. This is good common sense. And there is other work that you can put off because you don’t have to worry about it – you have to do it, but you know what you need to do and you know that you can do it. You might as well wait until you have fleshed out more of the system first so that you can get it done quickly. When should you do decide sooner rather than later? The kind of work that you can afford to put off safely is mostly obvious. And the kind of work that you can’t put off to later is also obvious, like figuring out the basic architecture, the platforms and languages that you are going to use, the basic approach that you are going to follow, proving out your core ideas. These are decisions that you need to make early on, while there is still time to learn and while there is still time to change your mind and start again if you have to. That’s why I was surprised by the example that Mike Cohn uses in his book: “Adding support for internationalization today may take four weeks of effort; adding it in six months may take six weeks. So we should add it now, right? Maybe. Suppose we spend four weeks and do it now. Over the next six months, we may spend an additional three weeks changing the original implementation based on knowledge gained during that six months. In that case, we would have been better off waiting. Or what if we spend four weeks now and later discover that a simpler and faster implementation would have been adequate?” But internationalization is a cross-cutting concern. This kind of work is often much harder to take care of than you want it to be or think it will be, impacting more of the design and code than you expect. Do you really want to wait six months before you find out that you have to change all of the forms and reports that you have developed to handle LTR and RTL languages simultaneously, or properly take care of laying out Japanese and Chinese characters? I’ve spent several weeks working with internationalization experts just trying to understand and solve problems with rendering honorific marks in Arabic names (a small thing, but a deal-killer if you are working with Middle Eastern royalty). Other cross-cutting concerns may require much more fundamental and more expensive changes to implement later, changes that are hard to get right and hard to test: data validation, security, monitoring and instrumentation, caching, logging, error handling, persistence. Understanding this upfront and supporting it from the start is much less risky and less costly than trying to “add it in later”. When in doubt, sooner is better than later Deciding to do something early means that you are probably going to make mistakes and you will have to throw some work away, and you’re going to have to make changes later when you get more information. All of this is waste. When it comes to planning out details, putting off decisions to just-in-time makes sense. But when it comes to designing and implementing important parts of the system, especially technical plumbing and cross-cutting work, doing something sooner is almost always better than later. Managing risk upfront will save you more money in the end. Reference: Sooner or Later: Deliver Early or Minimize Waste from our JCG partner Jim Bird at the Building Real Software blog....

Maven best practices

Although Maven offers a “convention over configuration” solution, there is still more then enough configuration necessary to cause some serious headache. In this post I will share some best practices with you that will ease maintenance of your POM files.Don’t use deprecated references like ${artifactId} or ${pom.artifactId}. Use the new ${project.artifactId} syntax. Note that this syntax follows the XML document structure, which makes it easy to remember and predict the value that the reference will result in. Try to avoid using inherited properties. Developers can easily forget that a certain property is used by a child POM and change the value breaking the build in an unexpected place. Secondly, its quite annoying not to be able to easily lookup a property without having to find and examine the parent POM. Use the dependencymanagement section of the parent pom to define all dependency versions, but do not set a scope here so that all dependencies have scope compile by default. Use properties to define the dependency versions. This way you can get an overview of all versions being used without having to scroll through multiple pages of dependency sections. Use the pluginmanagement section of the parent pom to define versions for *all* plugins that your build uses, even standard maven plugins like maven-compile-plugin and maven-source-plugin. This way your build will not suddenly behave differently when a new version of a plugin is released. When using a parent POM that is not located in the directory directly above the current POM define an empty relativePath element in your parent section. Use the dependency plugin to check your project for both unnecessary dependencies and undeclared-but-used-none-the-less dependencies. The goal is called ‘analyze’, so run the following command on the console: “mvn dependency:analyze” Make sure the pom files contain all the repository references needed to download all dependencies. If you want to use a local repository instead of downloadin strait from the internet then use the maven settings file to define mirrors for the individual repositories that are defined in the poms. If you use Nexus, then do not create repository groups containing both hosted and proxied repositories. This will dramaticly reduce the responsiveness because Nexus will check the remote locations of the proxied repositories even if a hosted repository contains the requested artifact.Make sure you read the best practices from Sonatype as well.Reference: Maven best practices from our JCG partner Geert Schuring at the Geert Schuring blog....

JBoss AS 7: Custom Login Modules

JBoss AS 7 is neat but the documentation is still quite lacking (and error messages not as useful as they could be). This post summarizes how you can create your own JavaEE-compliant login module for authenticating users of your webapp deployed on JBoss AS. A working elementary username-password module provided. Why to use Java EE standard authentication? Java EE security primer A part of the Java EE specification is security for web and EE applications, which makes it possible both to specify declarative constraints in your web.xml (such as “role X is required to access resources at URLs “/protected/*”) and to control it programatically, i.e. verifying that the user has a particular role (see HttpServletRequest.isUserInRole). It works as follows:You declare in your web.xml:Login configuration – primarily whether to use browser prompt (basic) or a custom login form and a name for the login realmThe custom form uses “magic” values for the post action and the fields, starting with j_, which are intercepted and processed by the serverThe roles used in your application (typically you’d something like “user” and perhaps “admin”) What roles are required for accessing particular URL patterns (default: none) Whether HTTPS is required for some parts of the applicationYou tell your application server how to authenticate users for that login realm, usually by associating its name with one of the available login modules in the configuration (the modules ranging from simple file-based user list to LDAP and Kerberos support). Only rarely do you need to create your own login module, the topic of this post.If this is new for you than I strongly recommend reading The Java EE 5 Tutorial – Examples: Securing Web Applications (Form-Based Authentication with a JSP Page incl. security constraint specification, Basic Authentication with JAX-WS, Securing an Enterprise Bean, Using the isCallerInRole and getCallerPrincipal Methods). Why to bother?Declarative security is nicely decoupled from the business code It’s easy to propagate security information between a webapp and for example EJBs (where you can protect a complete bean or a particular method declaratively via xml or via annotations such as @RolesAllowed) It’s easy to switch to a different authentication mechanism such as LDAP and it’s more likely that SSO will be supportedCustom login module implementation options If one of the login modules (a.k.a. security domains) provided out of the box with JBoss, such as UsersRoles, Ldap, Database, Certificate, isn’t sufficient for you then you can adjust one of them or implement your own. You can:Extend one of the concrete modules, overriding one or some of its methods to ajdust to your needs – see f.ex. how to override the DatabaseServerLoginModule to specify your own encryption of the stored passwords. This should be your primary choice, of possible. Subclass UsernamePasswordLoginModule Implement javax.security.auth.spi.LoginModule if you need maximal flexibility and portability (this is a part of Java EE, namely JAAS, and is quite complex)JBoss EAP 5 Security Guide Ch. 12.2. Custom Modules has an excellent description of the basic modules (AbstractServerLoginModule, UsernamePasswordLoginModule) and how to proceed when subclassing them or any other standard module, including description of the key methods to implement/override. You must read it. (The guide is still perfectly applicable to JBoss AS 7 in this regard.) The custom JndiUserAndPass module example, extending UsernamePasswordLoginModule, is also worth reading – it uses module options and JNDI lookup. Example: Custom UsernamePasswordLoginModule subclass See the source code of MySimpleUsernamePasswordLoginModule that extends JBoss’ UsernamePasswordLoginModule. The abstract UsernamePasswordLoginModule (source code) works by comparing the password provided by the user for equality with the password returned from the method getUsersPassword, implemented by a subclass. You can use the method getUsername to obtain the user name of the user attempting login. Implement abstract methods getUsersPassword() Implement getUsersPassword() to lookup the user’s password wherever you have it. If you do not store passwords in plain text then read how to customize the behavior via other methods below getRoleSets() Implement getRoleSets() (from AbstractServerLoginModule) to return at least one group named “Roles” and containing 0+ roles assigned to the user, see the implementation in the source code for this post. Usually you’d lookup the roles for the user somewhere (instead of returning hardcoded “user_role” role). Optionally extend initialize(..) to get access to module options etc. Usually you will also want to extend initialize(Subject subject, CallbackHandler callbackHandler, Map sharedState, Map options) (called for each authentication attempt),To get values of properties declared via the <module-option ..> element in the security-domain configuration – see JBoss 5 custom module example To do other initialization, such as looking up a data source via JNDI – see the DatabaseServerLoginModuleOptionally override other methods to customize the behavior If you do not store passwords in plain text (a wise choice!) and your hashing method isn’t supported out of the box then you can override createPasswordHash(String username, String password, String digestOption) to hash/encrypt the user-supplied password before comparison with the stored password. Alternatively you could override validatePassword(String inputPassword, String expectedPassword) to do whatever conversion on the password before comparison or even do a different type of comparison than equality. Custom login module deployment options In JBoss AS you canDeploy your login module class in a JAR as a standalone module, independently of the webapp, under <JBoss AS 7>/modules/, together with a module.xml – described at JBossAS7SecurityCustomLoginModules Deploy your login module class as a part of your webapp (no module.xml required)In a JAR inside WEB-INF/lib/ Directly under WEB-INF/classesIn each case you have to declare a corresponding security-domain it inside JBoss configuration (standalone/configuration/standalone.xml or domain/configuration/domain.xml): <security-domain name='form-auth' cache-type='default'> <authentication> <login-module code='custom.MySimpleUsernamePasswordLoginModule' flag='required'> <!--module-option name='exampleProperty' value='exampleValue'/--> </login-module> </authentication> </security-domain> The code attribute should contain the fully qualified name of your login module class and the security-domain’s name must match the declaration in jboss-web.xml: <?xml version='1.0' encoding='UTF-8'?> <jboss-web> <security-domain>form-auth</security-domain> <disable-audit>true</disable-audit> </jboss-web>The code Download the webapp jboss-custom-login containing the custom login module MySimpleUsernamePasswordLoginModule, follow the deployment instructions in the README. Reference: Creating Custom Login Modules In JBoss AS 7 (and Earlier) from our JCG partner Jakub Holy at the The Holy Java blog....

Servlet Basic Auth in an OSGi environment

You will first need to get a reference to the OSGI HTTP Service. You can do this through a declarative service. This post will concentrate on steps after getting a reference to the HTTP Service. Note: The complete class for this post is located here When registering a servlet through the OSGI HTTP Service, it provides you with an option to provide an implementation of HTTPContext.             httpService.registerServlet(alias, new MyServlet(), initParams, null);When we implement the HTTPContext interface we can implement three methods. Out of these three (3) handleSecurity will be called before serving a request to ermmm… check for security. public class BasicAuthSecuredContext implements HttpContext{ @Override public boolean handleSecurity(HttpServletRequest request, HttpServletResponse response) throws IOException { return false; }@Override public URL getResource(String s) { return null; }@Override public String getMimeType(String s) { return null; } }So, when implementing this, I’m borrowing a lot of content from the OSGI HTTPContext documentation and the HTTP Authentication spec. They are a must read, if you are interested in learning a lot, diving into details, etc. or you can just read the rest of this post. First, it is a big no-no to do basic auth unless https is used. If it’s not there we let the user know it’s forbidden territory. Let’s go ahead and do that. if (!request.getScheme().equals("https")) { response.sendError(HttpServletResponse.SC_FORBIDDEN); return false; }Next, let’s check for the Authorization header. If that’s not there we let them know, they need that shit to be there. Or we just say they’re unauthorized. Let’s do that now. if (request.getHeader("Authorization") == null) { response.sendError(HttpServletResponse.SC_UNAUTHORIZED); return false; }Ok, two tests passed. Now, we do some real work. Let’s extract the header decode it, and do “not so proper” authentication. protected boolean authenticated(HttpServletRequest request) { String authzHeader = request.getHeader("Authorization"); String usernameAndPassword = new String(Base64.decodeBase64(authzHeader.substring(6).getBytes()));int userNameIndex = usernameAndPassword.indexOf(":"); String username = usernameAndPassword.substring(0, userNameIndex); String password = usernameAndPassword.substring(userNameIndex + 1); // Now, do the authentication against in the way you want, ex: ldap, db stored uname/pw // Here I will do lame hard coded credential check. HIGHLY NOT RECOMMENDED! return ((username.equals("username") && password.equals("password")); }Let’s integrate this method, into the handleSecurity method. Notice how a meaningful error message is set to the response (line 14) when the security fails. This prevents the user from guessing, and they knows what exactly went wrong. Ermm, at least, if they know HTTP error codes they would know exactly what went wrong. @Override public boolean handleSecurity(HttpServletRequest request, HttpServletResponse response) throws IOException { if (!request.getScheme().equals("https")) { response.sendError(HttpServletResponse.SC_FORBIDDEN); return false; } if (request.getHeader("Authorization") == null) { response.sendError(HttpServletResponse.SC_UNAUTHORIZED); return false; } if (authenticated(request)) { return true; } else { response.sendError(HttpServletResponse.SC_UNAUTHORIZED); return false; } }That’s it. Now, pass this object when registering the servlet, httpService.registerServlet(alias, new MyServlet(), initParams, new BasicAuthSecuredContext());…and behold the power of basic auth in OSGI servlets! Reference: Implementing Basic Auth for a servlet in an OSGI environment from our JCG partner Mackie Mathew at the dev_religion blog....

Agile Estimating: Story Points and Decay

I’m re-reading Mike Cohn’s Agile Estimating and Planning. It’s the best book I’ve found on this and worth reading, even if he gets too Scrummy at times, and even if you don’t agree with everything he says. Which I don’t.For example, I don’t agree with him that Story Points are better for estimation than Ideal Days. When we do estimates, we use Ideal Days, because thinking this way is more natural and more useful, and because these estimates are more meaningful to people inside and outside of the team.  Estimates in Ideal Days DecayOne of the reasons that Cohn recommends Story Points is that, unlike estimates in Ideal Days, Story Point estimates “have a longer shelf life” because they don’t decay – the meanings of the estimates don’t change over time as the team gets more information.If you’re estimating using Ideal Days, as you learn more about the tools and language and platform, as you have more code to build on and work with, your understanding of how much work you can do in an Ideal Day will change. You may be able to get more work done – or less, because it turns out that you need to spend more time testing or whatever. As you learn and get better at how you work and understand it better, the definition of an “Ideal Day” will change. You’re learning and getting more information, so why not use this information and make better-quality decisions as you go forward?But this means the estimates that you did earlier aren’t as useful any more – they’ve decayed. You’ll have to re-estimate this work at some point, because you’re mixing before-the-fact and after-the-fact information.At some point you may have to do this anyways, whether you use Ideal Days or Story Points. Because relative-size estimates in Story Points also decay over time.  Story Points Decay tooEstimates in Ideal Days change as you learn more about the solution space: about how much work it actually takes to build some thing. Story Point estimates change as you learn more about the problem space: as you learn more about the domain and the problems that you need to solve, as you learn more about how big something really is in comparison to other things.Take Cohn’s “Zoo Points” exercise, where people attempt to estimate the relative size of different wild animals. Everyone knows that a blue whale is much bigger than a rabbit or a fox. But if you’ve never seen a blue whale in real life, you have no real idea just how much bigger it is. 10x? 100x? 500x? 1000x? What about a tarsier or an okapi – some of the rarest animals in the world. You’re unlikely to ever see one. How the heck could you know how big they are relative to a rabbit, or to each other? You don’t know, so you’ll need to make a reasonable guess and use that for the time being.Later, as you learn more about the problem space, especially if it’s a deeply technical domain, your understanding of size and complexity and risk will improve, and could change a lot.So your idea of whether a thing is twice as big as something else, or about 1/5 as big as another thing, whether it’s worth 5 points or 15 points, will change as you understand more about what these things actually are. As you continue to estimate the changes and new requirements that come in, your new estimates won’t align with earlier ones. The kind of work that you thought to be 5 points earlier you now know is 15 points. Unlike an Ideal Day, you haven’t changed what “5 points” means, but you have changed your understanding of what work is actually “5 points” worth.On a project that’s run for a long time, people will join the team bringing new knowledge and experience, and people will leave taking knowledge and experience with them. Which means that estimates that you made earlier, before you learned something, or when you knew something that you’ve now lost, can’t be the same as the estimates that you’re making now, regardless of whether you’re sizing work in days or points. It’s unavoidable – you can’t not use what you have learned, and you can’t keep using what you’ve lost.There may be reasons that Story Points are a better choice for estimating, for some teams at least, but “decay” isn’t one of them.Reference: Agile Estimating: Story Points and Decay from our JCG partner Jim Bird at the Building Real Software blog....
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below: