Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

and many more ....

Featured FREE Whitepapers

What's New Here?


Lambda Architecture for Big Data

An increasing number of systems are being built to handle the Volume, Velocity and Variety of Big Data, and hopefully help gain new insights and make better business decisions. Here, we will look at ways to deal with Big Data’s Volume and Velocity simultaneously, within a single architecture solution. Volume + Velocity Apache Hadoop provides both reliable storage (HDFS) and a processing system (MapReduce) for large data sets across clusters of computers. MapReduce is a batch query processor that is targeted at long-running background processes. Hadoop can handle Volume. But to handle Velocity, we need real-time processing tools that can compensate for the high-latency of batch systems, and serve the most recent data continuously, as new data arrives and older data is progressively integrated into the batch framework. Therefore we need both batch and real-time to run in parallel, and add a real-time computational system (e.g. Apache Storm) to our batch framework. This architectural combination of batch and real-time computation is referred to as a Lambda Architecture (λ). Generic Lambda λ has three layers:The Batch Layer  manages the master data and precomputes the batch views The Speed Layer  serves recent data only and increments the real-time views The Serving Layer is responsible for indexing and exposing the views so that they can be queried.The three layers are outlined in the below diagram along with a sample choice of technology stacks:Incoming data is dispatched to both Batch and Speed layers for processing. At the other end, queries are answered by merging both batch and real-time views. Note that real-time views are transient by nature and their data is discarded (making room for newer data) once propagated through the Batch and Serving layers. Most of the complexity is pushed onto the much smaller Speed layer where the results are only temporary, a process known as “complexity isolation“. We are indeed isolating the complexity of concurrent data updates in a layer that is regularly purged and kept small in size. λ is technology agnostic. The data pipeline is broken down into layers with clear demarcation of responsibilities, and at each layer, we can choose from a number of technologies. The Speed layer for instance could use either Apache Storm, or Apache Spark Streaming, or Spring “XD” ( eXtreme Data) etc. How do we recover from mistakes in λ ? Basically, we recompute the views. If that takes too long, we just revert to the previous, non-corrupted versions of our data. We can do that because of data immutability in the master dataset: data is never updated, only appended to (time-based ordering). The system is therefore Human Fault-Tolerant:  if we write bad data, we can just remove that data altogether and recompute. Unified Lambda The downside of λ is its inherent complexity. Keeping  in sync two already complex distributed systems is quite an implementation and maintenance challenge. People have started to look for simpler alternatives that would bring just about the same benefits and handle the full problem set. There are basically three approaches:Adopt a pure streaming approach, and use a flexible framework such as Apache Samza to provide some type of batch processing. Although its distributed streaming layer is pluggable, Samza typically relies on Apache Kafka. Samza’s streams are replayable, ordered partitions. Samza can be configured for batching, i.e. consume several messages from the same stream partition in sequence. Take the opposite approach, and choose a flexible Batch framework that would also allow micro-batches, small enough to be close to real-time, with Apache Spark/Spark Streaming or Storm’s Trident. Spark streaming is essentially a sequence of small batch processes that can reach latency as low as one second.Trident is a high-level abstraction on top of Storm that can process streams as small batches as well as do batch aggregation. Use a technology stack already combining batch and real-time, such as Spring “XD”, Summingbird or Lambdoop. Summingbird (“Streaming MapReduce”) is a hybrid system where both batch/real-time workflows can be run at the same time and the results merged automatically.The Speed layer runs on Storm and the Batch layer on Hadoop, Lambdoop (Lambda-Hadoop, with HBase, Storm and Redis) also combines batch/real-time by offering a single API for both processing paradigms:The integrated approach (unified λ) seeks to handle Big Data’s Volume and Velocity by featuring a hybrid computation model, where both batch and real-time data processing are combined transparently. And with a unified framework, there would be only one system to learn, and one system to maintain.Reference: Lambda Architecture for Big Data from our JCG partner Tony Sicilian at the Tony’s Blog blog....


When we think of regression, we think of bugs. That’s the first thing that pops into our minds. As with many other things, there’s a deeper meaning, if we just look closer. We have two kinds of usage for the word regression. We use it to describe a test suite that we run at the end of specific functional testing, to make sure everything is still working. We also use the term, technically, to describe an event when a test (either automated or manual) broke, when before it has passed. The first usage comes from the second; because we broke so much that had worked before, we needed to build a whole suite of tests to make sure nothing slipped by before the next release. We use the term “regression” to describe the system quality that has regressed. In short, something that worked before, now doesn’t. We have changed course, and instead of making progress, we are now regressing. If we want to move forward, or in any direction really, we don’t want to go back. Right? We don’t, when it comes to quality. We do, when the change is hard. Process regression So many times, we forget that regression is not just about bugs. When we’re making progress, with new processes or habits, every time we stumble, or hit a wall, we start thinking: It wasn’t that bad before. Sure, we had a lot of X problems (where X was the problem of the past, like people leaving because of command-and-control), but at least we didn’t need to deal with Y (where Y is the current challenge we’re facing, like losing control to a self-organizing team). When we do hit that wall, it’s very easy for us to regress. It makes sense, too. To make progress, we needed to get out of our comfort zone. When the going gets tough, it’s easy to get back there, because we feel comfortable there. It’s natural. It’s easier then climbing that wall. But here’s the thing: Before you start to regress, stop for a minute. Think of why you’ve started this journey in the first place. When you did, you wanted the prize at the end.  That prize (better quality, better results, less waste, any improvement you’re chasing) is still there. “But it’s a long, hard road!” Yes, it is. To make progress, you need to fight regression. Making progress is no simple matter.Reference: Regression from our JCG partner Gil Zilberfeld at the Geek Out of Water blog....

Apache Mesos : Writing your own distributed frameworks

In the previous post, we saw what mesos is, how is it useful and getting started with it. In this post, we shall see how to write your own framework on mesos. (In mesos, a framework is any application running on it.) This post explains about a framework called “mesos-pinspider” which fetches the user profile information and user board information of a pinterest page of a user. Mesos Framework In general, a Mesos framework has three basic components.    Driver which submits the tasks to the framework Scheduler which registers with the master to be offered resources, takes the tasks and runs them on executor Executor process that is launched on slave nodes to run the framework’s tasksPinspider Framework Example You may check the code here on github. Let’s break it down to PinDriver, PinScheduler and Pin UserProfileExecutor. Driver The driver component of the framework is PinDriver.Create Executor Info Describe the information about the executor using the Builder pattern and mesos use Google Protocol Buffers for the data interchange. Here, we need to set the executorID, command which is basically a shell command, executed via: ‘/bin/sh -c value’. Any URIs specified are fetched before executing the command. The name is set by setName(). The source is set by setSource(), an identifier style string used by frameworks to track the source of an executor. This is useful when it’s possible for different executor ids to be related semantically. Protos.ExecutorInfo userProfileExecutorInfo = Protos.ExecutorInfo.newBuilder().setExecutorId(Protos.ExecutorID.newBuilder().setValue("PinUserProfileExecutor")).setCommand(commandInfoUserProfile).setName("PinUserProfileExecutor Java").setSource("java").build();Create Framework Info Describe the framework information. The user field is used to determine the Unix user that an executor/task should be launched as. If the user field is set to an empty string Mesos will auto-magically set it to the current user. The amount of time that the master will wait for the scheduler to fail-over before removing the framework is specified by setFailoverTimeout(). The name of the framework is set by setName() Protos.FrameworkInfo.Builder frameworkBuilder = Protos.FrameworkInfo.newBuilder().setFailoverTimeout(120000).setUser("").setName("Pinspider Framework");Instantiate Scheduler You need to instantiate the Scheduler with the number of tasks that needs to be submitted for the executor to run.Scheduler scheduler = args.length == 1 ? new PinScheduler(userProfileExecutorInfo,userBoardExecutorInfo) : new PinScheduler(userProfileExecutorInfo, userBoardExecutorInfo, Integer.parseInt(args[1]), args[2]);Note: Please note that two ExecutorInfo are used ie. one for fetching user profile information and the other one for user board information for demonstration. This explanation involves only one executorinfo – userProfileExecutorInfo Starting the mesos scheduler driver. MesosSchedulerDriver is an implementation of SchedulerDriver which is an abstract interface to connect scheduler to mesos. This is done by managing the life-cycle of the scheduler ( start, stop and wait for tasks to finish) and also to interact with Mesos (launch tasks, kill tasks etc).MesosSchedulerDriver schedulerDriver = new MesosSchedulerDriver(scheduler,, args[0]);int status = == Protos.Status.DRIVER_STOPPED ? 0 : 1;schedulerDriver.stop();System.exit(status);Executor Implementation The Executor component of the framework is PinUserProfileExecutor. Executor is a callback interface which is implemented by frameworks’ executors. In our implementation, let us concentrate on launchTask()@Override public void launchTask(final ExecutorDriver executorDriver final Protos.TaskInfo taskInfo) {}Set the task status by setting the ID and the state with a builder pattern.Protos.TaskStatus taskStatus = Protos.TaskStatus.newBuilder().setTaskId(taskInfo.getTaskId()) .setState(Protos.TaskState.TASK_RUNNING).build();Send the status update to the framework scheduler retrying as necessary until an acknowledgement has been received or the executor is terminated, in which case, a TASK_LOST status update will be sent. executorDriver.sendStatusUpdate(taskStatus);Get the data from the tasks and run your logic.try { message = ("userprofile :" + getUserProfileInfo(url)).getBytes(); } catch (IOException e) { LOGGER.error("Error parsing the Pinterest URL :" + e.getMessage()); }Send the framework the message.executorDriver.sendFrameworkMessage(message);Mark the state of the task as finished and send the status update to the framework scheduler.taskStatus = Protos.TaskStatus.newBuilder().setTaskId(taskInfo.getTaskId()) .setState(Protos.TaskState.TASK_FINISHED).build(); executorDriver.sendStatusUpdate(taskStatus);main() method to create an instance of MesosExecutorDriver and run == Protos.Status.DRIVER_STOPPED ? 0 : 1Scheduler Implementation The Scheduler component of the framework is Pin Scheduler. Scheduler is a callback interface to be implemented by frameworks’ schedulers. In our implemenation, let us concentrate on resourceOffers(), statusUpdate() and frameworkMessage()Constructor : construct with the executor information and the number of launch tasks.public PinScheduler(Protos.ExecutorInfo pinUserProfileExecutor , Protos.ExecutorInfo pinUserBoardExecutor ) { this(pinUserProfileExecutor,pinUserBoardExecutor, 5, ""); }public PinScheduler(Protos.ExecutorInfo pinUserProfileExecutor,Protos.ExecutorInfo pinUserBoardExecutor, int totalTasks, String url) {this.pinUserProfileExecutor = pinUserProfileExecutor; this.pinUserBoardExecutor = pinUserBoardExecutor; this.totalTasks = totalTasks; this.crawlQueue = Collections.synchronizedList(new ArrayList<String>()); this.crawlQueue.add(url); }Resource OffersA resource offer can be resources like CPU, memory etc. From the offers list, get the scalar value of the resources. We need to give our requirements of resources for the tasks while setting the task info.for (Protos.Offer offer : list) { List<Protos.TaskInfo> taskInfoList = new ArrayList<Protos.TaskInfo>(); double offerCpus = 0; double offerMem = 0; for (Protos.Resource resource : offer.getResourcesList()) { if (resource.getName().equals("cpus")) { offerCpus += resource.getScalar().getValue(); } else if (resource.getName().equals("mem")) { offerMem += resource.getScalar().getValue(); } }"Received Offer : " + offer.getId().getValue() + " with cpus = " + offerCpus + " and mem =" + offerMem);Create task ID. Protos.TaskID taskID = Protos.TaskID.newBuilder().setValue(Integer.toString(launchedTasks++)).build();Create task info by setting task ID, adding resources, setting data and setting executor.Protos.TaskInfo pinUserProfileTaskInfo = Protos.TaskInfo.newBuilder().setName("task " + taskID.getValue()) .setTaskId(taskID).setSlaveId(offer.getSlaveId()) .addResources(Protos.Resource.newBuilder().setName("cpus") .setType(Protos.Value.Type.SCALAR) .setScalar(Protos.Value.Scalar .newBuilder().setValue(CPUS_PER_TASK))) .addResources(Protos.Resource.newBuilder().setName("mem") .setType(Protos.Value.Type.SCALAR) .setScalar(Protos.Value.Scalar.newBuilder().setValue(MEM_PER_TASK))) .setData(ByteString.copyFromUtf8(crawlQueue.get(0))) .setExecutor(Protos.ExecutorInfo.newBuilder(pinUserProfileExecutor)).build();Launch the tasks through the SchedulerDriver. ... taskInfoList.add(pinUserProfileTaskInfo); taskInfoList.add(pinUserBoardTaskInfo); } schedulerDriver.launchTasks(offer.getId(), taskInfoList);Status update This is invoked when the status of a task has changed ie., a slave is lost and so the task is lost, a task finishes and an executor sends a status update saying so. @Override public void statusUpdate(SchedulerDriver schedulerDriver, Protos.TaskStatus taskStatus) { ... }Stop the SchedulerDriver if tasks are finishedif (taskStatus.getState() == Protos.TaskState.TASK_FINISHED) {finishedTasks++;"Finished tasks : " + finishedTasks);if (finishedTasks == totalTasks) {schedulerDriver.stop();}}Abort the SchedulerDriver if the tasks are killed, lost or failed if (taskStatus.getState() == Protos.TaskState.TASK_FAILED || taskStatus.getState() == Protos.TaskState.TASK_KILLED || taskStatus.getState() == Protos.TaskState.TASK_LOST) { LOGGER.error("Aborting because the task " + taskStatus.getTaskId().getValue() + " is in unexpected state : " + taskStatus.getState().getValueDescriptor().getName() + "with reason : " + taskStatus.getReason().getValueDescriptor().getName() + " from source : " + taskStatus.getSource().getValueDescriptor().getName() + " with message : " + taskStatus.getMessage()); schedulerDriver.abort(); }Framework Message This is invoked when an executor sends a message.Handle your message @Override public void frameworkMessage(SchedulerDriver schedulerDriver, Protos.ExecutorID executorID, Protos.SlaveID slaveID, byte[] bytes) { String data = new String(bytes); System.out.println(data);"User Profile Information : " + data); }Complete code is available here with the instructions to run and sample output.Reference: Apache Mesos : Writing your own distributed frameworks from our JCG partner Swathi V at the * Techie(S)pArK * blog....

How To Process Java Annotations

One of the cool new features of Java 8 is the support for lambda expressions. Lambda expressions lean heavily on the FunctionalInterface annotation. In this post, we’ll look at annotations and how to process them so you can implement your own cool features. Annotations Annotations were added in Java 5. The Java language comes with some predefined annotations, but you can also define custom annotations.     Many frameworks and libraries make good use of custom annotations. JAX-RS, for instance, uses them to turn POJOs into REST resources. Annotations can be processed at compile time or at runtime (or even both). At runtime, you can use the reflection API. Each element of the Java language that can be annotated, like class or method, implements the AnnotatedElement interface. Note that an annotation is only available at runtime if it has the RUNTIME RetentionPolicy. Compile-Time Annotation Processing Java 5 came with the separate apt tool to process annotations, but since Java 6 this functionality is integrated into the compiler. You can either call the compiler directly, e.g. from the command line, or indirectly, from your program. In the former case, you specify the -processor option to javac, or you use the ServiceLoader framework by adding the file META-INF/services/javax.annotation.processing.Processor to your jar. The contents of this file should be a single line containing the fully qualified name of your processor class. The ServiceLoader approach is especially convenient in an automated build, since all you have to do is put the annotation processor on the classpath during compilation, which build tools like Maven or Gradle will do for you. Compile-Time Annotation Processing From Within Your Application You can also use the compile-time tools to process annotations from within your running application. Rather than calling javac directly, use the more convenient JavaCompiler interface. Either way, you’ll need to run your application with a JDK rather than just a JRE. The JavaCompiler interface gives you programmatic access to the Java compiler. You can obtain an implementation of this interface using ToolProvider.getSystemJavaCompiler(). This method is sensitive to the JAVA_HOME environment variable. The getTask() method of JavaCompiler allows you to add your annotation processor instances. This is the only way to control the construction of annotation processors; all other methods of invoking annotation processors require the processor to have a public no-arg constructor. Annotation Processors A processor must implement the Processor interface. Usually you will want to extend the AbstractProcessor base class rather than implement the interface from scratch. Each annotation processor must indicate the types of annotations it is interested in through the getSupportedAnnotationTypes() method. You may return * to process all annotations. The other important thing is to indicate which Java language version you support. Override the getSupportedSourceVersion() method and return one of the RELEASE_x constants. With these methods implemented, your annotation processor is ready to get to work. The meat of the processor is in the process() method. When process() returns true, the annotations processed are claimed by this processor, and will not be offered to other processors. Normally, you should play nice with other processors and return false. Elements and TypeMirrors The annotations and the Java elements they are present on are provided to your process() method as Element objects. You may want to process them using the Visitor pattern. The most interesting types of elements are TypeElement for classes and interfaces (including annotations), ExecutableElement for methods, and VariableElement for fields. Each Element points to a TypeMirror, which represents a type in the Java programming language. You can use the TypeMirror to walk the class relationships of the annotated code you’re processing, much like you would using reflection on the code running in the JVM. Processing Rounds Annotation processing happens in separate stages, called rounds. During each round, a processor gets a chance to process the annotations it is interested in. The annotations to process and the elements they are present on are available via the RoundEnvironment parameter passed into the process() method. If annotation processors generate new source or class files during a round, then the compiler will make those available for processing in the next round. This continues until no more new files are generated. The last round contains no input, and is thus a good opportunity to release any resources the processor may have acquired. Initializing and Configuring Processors Annotation processors are initialized with a ProcessingEnvironment. This processing environment allows you to create new source or class files. It also provides access to configuration in the form of options. Options are key-value pairs that you can supply on the command line to javac using the -A option. For this to work, you must return the options’ keys in the processor’s getSupportedOptions() method. Finally, the processing environment provides some support routines (e.g. to get the JavaDoc for an element, or to get the direct super types of a type) that come in handy during processing. Classpath Issues To get the most accurate information during annotation processing, you must make sure that all imported classes are on the classpath, because classes that refer to types that are not available may have incomplete or altogether missing information. When processing large numbers of annotated classes, this may lead to a problem on Windows systems where the command line becomes too large (> 8K). Even when you use the JavaCompiler interface, it still calls javac behind the scenes. The Java compiler has a nice solution to this problem: you can use argument files that contain the arguments to javac. The name of the argument file is then supplied on the command line, preceded by @. Unfortunately, the JavaCompiler.getTask() method doesn’t support argument files, so you’ll have to use the underlying run() method. Remember that the getTask() approach is the only one that allows you to construct your annotation processors. If you must use argument files, then you have to use a public no-arg constructor. If you’re in that situation, and you have multiple annotation processors that need to share a single instance of a class, you can’t pass that instance into the constructor, so you’ll be forced to use something like the Singleton pattern. Conclusion Annotations are an exciting technology that have lots of interesting applications. For example, I used them to extract the resources from a REST API into a resource model for further processing, like generating documentation.Reference: How To Process Java Annotations from our JCG partner Remon Sinnema at the Secure Software Development blog....

How to Be Honest and Keep a Customer

Most of our clients are rather surprised when we explain to them that they will have full access to the source code from the first day of the project. We let them see everything that is happening in the project, including the Git repository, bug reports, discussions between programmers, continuous integration fails, etc. They often tell me that other software development outsourcing teams keep this information in-house and deliver only final releases, rarely together with the source code. I understand why other developers are trying to hide as much as possible. Giving a project sponsor full access to the development environment is not easy at all. Here is a summary of problems we’ve been having and our solutions. I hope they help you honestly show your clients all project internals and still keep them on board.He Is Breaking Our Process This is the most popular problem we face with our new clients. Once they gain access to the development environment, they try to give instructions directly to programmers, walking around our existing process. “I’m paying these guys; why can’t I tell them what to do?” is a very typical mindset. Instead of submitting requests through our standard change management mechanism, such a client goes directly to one of the programmers and tells him what should be fixed, how, and when. It’s micro-management in its worst form. We see it very often. What do we do? First, we try to understand why it’s happening. The simplest answer is that the client is a moron. Sometimes this is exactly the case, but it’s a rare one. Much more often, our clients are not that bad. What is it, then? Why can’t they follow the process and abide by the rules? There are a few possible reasons. Maybe the rules are not explained well. This is the most popular root cause — the rules of work are not clear enough for the client. He just doesn’t know what he is supposed to do in order to submit a request and get it implemented. To prevent this, we try to educate our clients at the beginning of a new project. We even write guidance manuals for clients. Most of them are happy to read them and learn the way we work, because they understand that this is the best way to achieve success while working with us. Maybe our management is chaotic, and the client is trying to “organize” us by giving explicit instructions regarding the most important tasks. We’ve seen it before, and we are always trying to learn from this. As soon as we see that the client is trying to micro-manage us, we ask ourselves: “Is our process transparent enough? Do we give enough information to the client about milestones, risks, plans, costs, etc.?” In most cases, it’s our own fault, and we’re trying to learn and improve. If so, it’s important to react fast, before the client becomes too agressive in his orders and instructions. It will be very difficult to escort him back to the normal process once he gets “micro-management” in his blood. Maybe the client is not busy enough and has a lot of free time, which he is happy to spend by giving orders and distracting your team. I’ve seen this many times. A solution? Keep him busy. Turn him into a member of the team and assign him some tasks related to documentation and research. In my experience, most clients would be happy to do this work and help the project. He Is Asking Too Much A technically-savvy client can turn the life of an architect into a nightmare by constantly asking him to explain every single technical decision made, from “Why PostgreSQL instead of MySQL?” to “Why doesn’t this method throw a checked exception?” Constantly answering such questions can turn a project into a school of programming. Even though he is paying for our time, that doesn’t mean we should teach him how to develop software, right? On the other hand, he is interested in knowing how his software is developed and how it works. It’s a fair request, isn’t it? I believe there is a win-win solution to this problem. Here is how we manage it. First of all, we make all his requests formal. We ask a client to create a new ticket for each request, properly explaining what is not clear and how much detail is expected in the explanation. Second, we look at such requests positively — they are good indicators of certain inconsistencies in the software. If it’s not clear for the client why PostgreSQL is used and not MySQL, it’s a fault of our architect. He didn’t document his decision and didn’t explain how it was made, what other options were considered, what selection criteria were applied, etc. Thus, a request from a client is a bug we get for free. So, we look at it positively. Finally, we charge our clients for the answers given. Every question, submitted as a ticket, goes through the full flow and gets billed just as any other ticket. This approach prevents the client from asking for too much. He realizes that we’re ready to explain anything he wants, but he will pay for it. He Is Telling Too Much This problem is even bigger than the previous one. Some clients believe they are savvy enough to argue with our architect and our programmers about how the software should be developed. They don’t just ask why PostgreSQL is used, they tell us that we should use MySQL, because “I know that it’s a great database; my friend is using it, and his business is growing!” Sometimes it gets even worse, when suggestions are directed at every class or even a method, like “You should use a Singleton pattern here!” Our first choice is to agree and do what he wants. But it’s a road to nowhere. Once you do it, your project is ruined, and you should start thinking about a divorce with this client. Your entire team will quickly turn into a group of coding monkeys, micro-managed by someone with some cash. It’s a very wrong direction; don’t even think about going there. The second choice is to tell the client to mind his own business and let us do ours. He hired us because we’re professional enough to develop the software according to his requirements. If he questions our capabilities, he is free to change the contractor. But until then, he has to trust our decisions. Will this work? I doubt it. It’s the same as giving him the finger. He will get offended, and you won’t get anything. The solution here is to turn the client’s demands into project requirements. Most of them will be lost in the process, because they won’t be sane enough to form a good requirement. Others will be documented, estimated, and crossed-out by the client himself, becuase he will realize they are pointless or too expensive. Only a few of them will survive, since they will be reasonable enough. And they will help the project. So it is also a win-win solution. For example, he says that “you should use MySQL because it’s great”. You tell him that the project requirements document doesn’t limit you to choose whichever database you like. Should it? He says yes, of course! OK, let’s try to document such a requirement. How will it sound? How about, “We should only use great databases?” Sound correct? If so, then PostgreSQL satisfies this requirement. Problem solved; let us continue to do our work. He will have a hard time figuring out how to write a requirement in a way that disallows PostgreSQL but allows MySQL. It is simply not possible in most cases. Sometimes, though, it will make sense; for example, “We should use a database server that understands our legacy data in MySQL format”. This is a perfectly sane requirement, and the only way to satisfy it is to use MySQL. Thus, my recommendation is to never take a client’s demands directly to execution, but rather use them first to amend the requirements documentation. Even if you don’t have such documentation, create a simple one-page document. Agree with the client that you work against this document, and when anyone wants to change something, you first have to amend the document and then have your team ensure the software satisfies it. This kind of discipline will be accepted by any client and will protect you against sudden and distracting corrections. He Is Questioning Our Skills When source code is open to the client, and he is technically capable of reading it, it is very possible that one day he will tell us that our code is crap and we have to learn how to program better. It has not happened in our projects for many years, but it has happened before, when we weren’t using static analysis as a mandatory step in our continuous integration pipeline. Another funny possibility is when the client shows the source code to a “friend”, and he gives a “professional” opinion, which sounds like, “They don’t know what they are doing.” Once such an opinion hits your client’s ears, the project is at a significant risk of closure. It’ll be very difficult, almost impossible, to convince the client not to listen to the “friend” and continue to work with you. That’s why most outsourcers prefer to keep their sources private until the very end of the project, when the final invoice is paid. I think that an accidental appearance of a “friend” with a negative opinion is un-preventable. If it happens, it happens. You can’t avoid it. On the other hand, if you think your code is perfect and your team has only talented programmers writing beautiful software, this is not going to protect you either. An opinion coming from a “friend” won’t be objective; it will just be very personal, and that’s why it’s very credible. He is a friend of a client, and he doesn’t send him bills every week. Why would he lie? Of course, he is speaking from the heart! (I’m being sarcastic.) So, no matter how beautiful your architecture and your source code is, the “friend” will always be right. In my opinion, the only way to prevent such a situation or minimize its consequences is to organize regular and systematic independent technical reviews. They will give confindence to the client that the team is not lying to him about the quality of the product and key technical decisions made internally. To conclude, I strongly believe it is important to be honest and open with each client, no matter how difficult it is. Try to learn from every conflict with each client, and improve your management process and your principles of work. Hiding source code is not professional and makes you look bad in the eyes of your clients and the entire industry.Reference: How to Be Honest and Keep a Customer from our JCG partner Yegor Bugayenko at the About Programming blog....

Notes: Microservices by Martin Fowler

I just watched Martin Fowler’s talk about Microservices at XConf. Here are some of my notes and related thoughts. The talk is heavily based upon Martin Fowler and James Lewis’ article about Microservices, so you can find and read most of the information there.Characteristics of Microservices In his article, Fowler defines some characteristics that most of the already existing Microservices architectures implement. In his talk he discusses some of them: Componentization via Services - What really is a component? The term often pops up as a “cooler” alternative to module or widget. But there’s a vague definition: is it a class, an executable piece? From Fowler’s perspective it is best to use some real world paradigm, like how a user refers to a “stereo system”. They move it around, attach it somewhere else. Move, remove and attach different speakers. Basically a component is something that is independently upgradeable and replaceable. There are major ways about how components are being used or brought into a project: through libraries (jars, gems, node_modules,…) or through services. The difference is simply that services are not directly brought into your codebase, but rather be called through remote calls. Organized around business capabilities - The organization of teams and people is a very important point. Many organization structure their teams around technology: UI specialists, middleware team, DBAs. Rather, with microservices, people should be organized around the business capabilities within cross-functional teams: like the “shipping team”, the “orders team”,…According to Amazon these teams should be as big s.t. we’re able to feed them with 2 (American) pizzas (a dozen of people). Fowler highlights that an important fact is that these teams have a direct communication line to the end user or customer and get according feedback how the stuff they build is being used and how well or not it works. Microservices is much more about about team organization rather than software architecture. Architecture and team organization is always heavily coupled together. organizations which design systems….are constrained to produce designs which are copies of the communication structures of these organizations – Conway’s law Smart endpoints and dumb pipes - It is common practice to use smart network infrastructure like ESBs that contain logic about how to treat certain network messages and how to route them around. Microservices instead facilitate dumb pipes and smart endpoints/applications. The problem is that smart pipes (i.e. ESBs) lead to issues with continuous delivery as they cannot be easily version controlled or integrated into a large pipeline. Moreover it creates dependencies with the application itself, meaning that when you decide to upgrade your endpoint/service, you often have to do some work on the ESB as well. Decentralized Data Management - Normally, on a monolith system there’s one huge database where all the data is being stored. Often there are even multiple monoliths connecting to the same DB. In a service oriented approach, each service gets its own database and the data is not shared directly with others. Sharing has to go through service that wraps the data. This leads to a lot of beneficial flexibility on the service’s side as it can decide which technology to adopt, which DBMS system etc. In this way, different programming languages and database systems can be used among services. It’s decentralized decision making. Infrastructure Automation - Continuous delivery is a must have as well as automated mechanisms for machines provisioning, for deployment, testing etc. Design for failure - You inevitably have to design for failure as the microservices will fail, even frequently. Netflix is renowned for carrying this to the extreme. They have a “Chaos Monkey” which runs over their production system during the day and randomly shuts down services. This gives them valuable feedback about how resistant their services are, how well they recover from outages and how the overall system is affected. Although many people tend to abstract and hide remote calls, you cannot expect them to perform like normal calls. Expect them to fail rather than to succeed. A notable library released by Netflix is Hysterix, a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries. Others characteristics that haven’t been mentioned explicitly in the talk are these:Products not Projects Decentralized Governance Infrastructure Automation Evolutionary DesignYou find them explained in the microservice article on Fowler’s website. Are Microservices really just SOA? A question I’ve posed myself as well a couple of times. The point is – and Fowler mentions that as well – is how one defines SOA. If SOA is being seen as ESBs, smart centralized management,… In that case Microservices is very different. Fowler defines it as a subset of SOA, one that refers to a particular style of implementing certain concepts. Having “micro” in its name, the question arises about how big a service should be? Actually there’s a lot of variation, from 15 people -> 10 services to 4 people -> 200 services. It’s not yet exactly clear. When to use a Monolith and when a Microservice? Fowler immediately emphasizes that with every new concept people tend to overuse it. So be cautious about that. Monolithsimplicity: take into account that distributed services are a huge complexity booster consistency inter-module refactoring is much easier. Within a microservice architecture, if you fail to do a good service-cut it’s much more difficult to move logic around.MicroservicesPartial deployment Good degree of availability (if good failure management has been implemented) Preserve modularity: programming languages are bad in helping to preserve the modularity of systems as it is too easy to escape their structures (like packages and namespaces) Multiple platforms as such architecture gives more flexibility in the technology choices such as OS, programming language etc…You must be this tall… …to use microservices. As already mentioned in Fowler’s article about the prerequisites you should fulfill the following prerequisites:Fast provisioning: be able to setup new machines in hours not days in order to avoid delays. Good monitoring: to be quickly able to diagnose failures and trace them down to the place they occurred. If issues occur you should be able to quickly rollback to a previous version of the microservice. Rapid Application deployment through fully automated pipelines Devops cultureThat said, Fowler says that when he’s not sure he still prefers the Monolith structure and eventually evolve to a Microservices system later in the development cycle (although that might not be totally optimal). Conclusion The talk somehow confirms my impressions so far…high degree of automation at various levels (infrastructure, testing, dev, deployment) trend to empower the team, its responsibility and flexibility in the decision making (compatible with what Agile already fosters) implementing best practices (fault tolerance, smart endpoints, monitoring, testing) know the trade-offs, don’t overuseAs an architect and tech enthusiast this is the architecture of my dreams. But given the issues I regularly face when coaching teams, the difficulty at adopting good programming practices and rigid organizational structures…then adopting a Microservices architecture is extremely difficult. Still, even if you’re not ready yet, I think you should aim for such structure, without necessarily going it the whole way. Like, structuring your application or product by following practices from the Microservices architecture but deploying it as a monolith. I’m thinking about a modular architecture, good failure handling, logging and monitoring, automated deployment pipelines and a high degree of automated testing. From there, optimize incrementally.Reference: Notes: Microservices by Martin Fowler from our JCG partner Juri Strumpflohner at the Juri Strumpflohner’s TechBlog blog....

Using Netflix Hystrix annotations with Spring

I can’t think of a better way to describe a specific feature of Netflix Hystrix library than by quoting from its home page:Latency and Fault Tolerance by: Stop cascading failures. Fallbacks and graceful degradation. Fail fast and rapid recovery. Thread and semaphore isolation with circuit breakers. I saw a sample demonstrated by Josh Long(@starbuxman) which makes use of Hystrix integrated with Spring – the specific code is here. The sample makes use of annotations to hystrix enable a service class. My objective here is to recreate a similar set-up in a smaller unit test mode. With that in mind, consider the following interface which is going to be made fault tolerant using Hystrix library: package hystrixtest;public interface RemoteCallService {String call(String request) throws Exception;} And a dummy implementation for it. The dummy implementation delegates to a mock implementation which in-turn fails the first two times it is called and succeeds with the third call: package hystrixtest;import; import org.mockito.invocation.InvocationOnMock; import org.mockito.stubbing.Answer;import static org.mockito.Mockito.*;public class DummyRemoteCallService implements RemoteCallService {private RemoteCallService mockedDelegate;public DummyRemoteCallService() { try { mockedDelegate = mock(RemoteCallService.class); when( .thenThrow(new RuntimeException("Deliberately throwing an exception 1")) .thenThrow(new RuntimeException("Deliberately throwing an exception 2")) .thenAnswer(new Answer<String>() { @Override public String answer(InvocationOnMock invocationOnMock) throws Throwable { return (String) invocationOnMock.getArguments()[0]; } }); }catch(Exception e) { throw new IllegalStateException(e); } }@Override @HystrixCommand(fallbackMethod = "fallBackCall") public String call(String request) throws Exception { return; }public String fallBackCall(String request) { return "FALLBACK: " + request; } } The remote call has been annotated with the @Hystrixcommand annotation with a basic configuration to fall back to a “fallBackCall” method in case of a failed remote call. Now, as you can imagine, there has to be something in the Hystrix library which should intercept calls annotated with @HystrixCommand annotation and makes it fault tolerant. This is a working test which wraps the necessary infrastructure together – in essence, Hystrix library provides a companion AOP based library that intercepts the calls. I have used Spring testing support here to bootstrap the AOP infrastructure, to create the HystrixCommandAspect as a bean, the call goes to the “fallBackCall” for the first two failed calls and succeeds the third time around: package hystrixtest;import; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.EnableAspectJAutoProxy; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;import static org.hamcrest.MatcherAssert.assertThat; import static;@RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration public class TestRemoteCallServiceHystrix {@Autowired private RemoteCallService remoteCallService ;@Test public void testRemoteCall() throws Exception{ assertThat("test"), is("FALLBACK: test")); assertThat("test"), is("FALLBACK: test")); assertThat("test"), is("test")); }@Configuration @EnableAspectJAutoProxy public static class SpringConfig {@Bean public HystrixCommandAspect hystrixCommandAspect() { return new HystrixCommandAspect(); }@Bean public RemoteCallService remoteCallService() { return new DummyRemoteCallService(); } } } Spring-Cloud provides an easier way to configure the Netflix libraries for Spring-Boot based projects and if I were to use this library the test would transform to this, a bunch of configuration is now commented out with the help of Spring-Boot: package hystrixtest;import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.autoconfigure.EnableAutoConfiguration; import org.springframework.boot.test.SpringApplicationConfiguration; import; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;import static org.hamcrest.MatcherAssert.assertThat; import static;@RunWith(SpringJUnit4ClassRunner.class) @SpringApplicationConfiguration public class TestRemoteCallServiceHystrix {@Autowired private RemoteCallService remoteCallService;@Test public void testRemoteCall() throws Exception { assertThat("test"), is("FALLBACK: test")); assertThat("test"), is("FALLBACK: test")); assertThat("test"), is("test")); }@Configuration @EnableAutoConfiguration // @EnableAspectJAutoProxy @EnableHystrix public static class SpringConfig {// @Bean // public HystrixCommandAspect hystrixCommandAspect() { // return new HystrixCommandAspect(); // }@Bean public RemoteCallService remoteCallService() { return new DummyRemoteCallService(); } } } If you are interested in exploring this sample further, here is the github repo with the working tests.Reference: Using Netflix Hystrix annotations with Spring from our JCG partner Biju Kunjummen at the all and sundry blog....

How to perform client side web performance testing in JMeter?

In this article we will see how can we do client side performance testing using Jmeter Plugins. I will be using jmeter webdriver plugins. Before starting this topic, please have some basic information on client side performance testing from my previous post. So, lets get started:         InstallationInstall Jmeter and Plug ins from below links following this post.Jmeter Plugins (you may choose only web driver, but I prefer all of them)Download Selenium Server from here. (you need java to run that) Download Firefox 26 from Archive. Why 26? because jmeter webdriver plugins supports firefox 26. Here is link where you see supports the details. Note: This can be tricky if you have an updated Firefox version. In that case you can do like me.Disable firefox update checking Install in new folder in separate directory name.When you run this one for the first time, just cancel the initial update process. As you have disabled firefox update (in your updated firefox), make sure you see the update settings disabled in this firefox 26 too.Note: This part is little tricky, I have provided separate post to resolve it. For Jmeter Remote Execution or local, it is better to to have only one firefox (version 26) with no auto update settings which will minimize complexity for test execution. Keep firefox 26, selenium server in path variable. To check, type firefox from command line and run. You should see, firefox 26 launched in desktop.Setting Jmeter: Usually, we do not need any extra things for webdriver sampler. But, as we need debugging, we might use this following property in file.It enables sub sampling which is good for debugging. webdriver.sampleresult_class=true Let me explain how it works: JMeter webdriver sampler is just an extension of http sampler, not alternative, with a script editor. When it runs, it actually calls firefox driven by webdriver. That means, it sends its instruction to mainly webdriver and, webdriver does every thing. Now, you might be wondering how code goes to the web driver. Like as other code support, webdriver core runs as external code following JSR specification. It is actually JavaScript execution. And, you see, it is just like as webdriver java code with some basic modification due to jmeter adoption. I will provide separate blog or coding. And after you write the steps as webdriver script, use listeners to get time. Like as other samplers, you use listeners to debug sensibly.Browser supporting Just follow this link which mentions the configurable browser names that are supported by webdriver sampler. You can see this from jmeter too:Time Measurement Webdriver sampler calculates time from this line of code: WDS.sampleResult.sampleStart() to this line of code: WDS.sampleResult.sampleEnd() So, for debugging, we need sub samples which will be shown as child of main sample. For this purpose, we need to activate sampleresult_class (set to true). After activation we can do sub sampling like WDS.sampleResult.sampleStart() //Brows to a URL //Add user name and password WDS.sampleResult.subSampleStart('Log In request') //Click Log in WDS.sampleResult.subSampleEnd(true) //Check title //do some another process WDS.sampleResult.sampleEnd() In view result tree listener, you can see the main sample that contains the sub sample name “’Log In request’”. And a main sample can have multiple sub sample. That means, we can calculate each time separately from result tree. Note that sub samples will not be shown separately in tabular format listeners or graphs. And, if we need to measure particular transaction, we can split single test among multiple webdriver samples. For example, a sampler for log in, a sampler for doing some work in home page, a sampler for messaging, etc.. In this way, we can see results in reports. Usually each business transaction is measured in separate sample where as detail steps are sub sampled. Writing the first script: To write a webdriver sampler script, you need to add the webdriver sampler (sampler –>webdriver sampler) with any browser (driver) configuration. See image from the browser support section to get the driver configuration elements. I will provide a separate post on how to write a webdriver test script with an example. You can see some nice guidelines at the wedriver sampler wiki. Test Planning: As we know from my previous post on client side performance test, this test should run from a single user or thread. As the jmeter sampler populates the browser with webdriver, this has a particular hardware requirement. That is, it will occupy a single thread of a processor. That means, if you want to run the webdriver sampler, you need at least 2 core CPU. Why 2 core? The other one is for Jmeter. So, if you have 8 core CPUs, you can run only 7 threads for webdriver samplers. So, for testing we have to add a separate thread group or a test block for webdriver sampler. We will run for measuring client execution time on:When there is not much user load When average load on server When High load(considered as peak load)Sometimes, it is also good to test the following too:Beyond capacity, where error can be occurred or may just after error condition. As continuous performance testing. Usually, people run selected regression test with jmeter daily or weekly.Again, the formula is simple, 1 thread of CPU for single client performance testing. And, you run the test simply as Jmeter test, that’s all.Reference: How to perform client side web performance testing in JMeter? from our JCG partner Shantonu Sarker at the A Test Developer’s blog blog....

How to get all Method information under a Class in Java Reflection?

This article is building on my previous post. In this article we will see how can we retrieve Class Related Information using Java Reflection. We will focus on Method Names. Note: I will make a separate reflector utility class where we input a target class in its constructor and we will retrieve information using a separate method. In this way, we can isolate our needs. Please see this before we start. How to get all declared Method Names inside of a class? This means, we will get the method names which are declared inside the class (public, private, default, protected), i.e. not inherited methods. public String[] getAllOwnMethodNames(){ArrayList<String> allMethods = new ArrayList<String>();for(Method aMethod : myClass.getDeclaredMethods()){allMethods.add("Method Name : "+aMethod.getName()+" , Full Name : "+aMethod.toString());}return allMethods.toArray(new String[allMethods.size()]);} How to get all Method Names accessible from of a class (which includes inherited, implemented methods of its own, super class, interfaces)? public String[] getAllPubliAndInheritedMethodNames(){ArrayList<String> allMethods = new ArrayList<String>();for(Method aMethod : myClass.getMethods()){allMethods.add("Method Name : "+aMethod.getName()+" , Full Name : "+aMethod.toString());}return allMethods.toArray(new String[allMethods.size()]);} Note: To have the information in detail we use the getName() and toString() methods. For both cases, we can specify the method name to get that specific method. myClass.getDeclaredMethod(<Name of the method as string>, parameter of that method) myClass.getMethod(<Name of the method as string>, parameter of that method) In both cases we need to know the name of the method. Sometimes, for a class, we need to know if a method is a Getter or a setter method. We can apply a small string filter, like the following: To know if it is a Getter method : aMethod.getName().startsWith("set"); To know if it is a Setter method : aMethod.getName().startsWith("get");How to get all Method information under a Class in Java Reflection? from our JCG partner Shantonu Sarker at the A Test Developer’s blog blog....

Dependency injection pitfalls in Spring

There are three injection variants in Spring framework:Setter-based injection Constructor-based injection Field-based injectionEach of those mechanisms has advantages and disadvantages and there is not only one right approach. For example field injection:       ...
Java Code Geeks and all content copyright © 2010-2015, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

Get ready to Rock!
To download the books, please verify your email address by following the instructions found on the email we just sent you.