Featured FREE Whitepapers

What's New Here?


Using IntelliJ..for 2 weeks, so far so good

It’s been almost 2 weeks that I have completely switched over to IntelliJ as my main Java IDE at home and at work. So far so good, here are my  initial findings.        Migration: I took me a couple of hours to migrated my projects over. Eventually if your project is already Mavenized, things are simple, no risk involved. Maven: As many people say, IntelliJ currently treats Maven-ized projects better, comparing to Eclipse Kepler and its internal plugin. The integration is not perfect, but I don’t thing there is such a thing. Profiles work , maven options work, the IDE seems to ‘re-fresh’ it’s state along with the ‘Maven’ one, especially during clean and package. This is what I wanted, so I am very happy about it. Key Bindings : At first I had selected the Eclipse Key Map, but soon realized that most of the examples out there were based on the intelliJ key bindings (especially when you were browsing help stuff). At the same time, some of the most exotic and clever functionality was not by default ‘configured’ to an eclipse combo. So I was feeling, I was missing some magic. During the second week, decided to change my settings to IntelliJ defaults and I was surprised that after a a day or so  with the help of the documentation and the Cmd+Shift+A, I found my way around.  Crashes : No crashes, ooh yes, this is so good. No crashes. Enterprise Features / Facets : I tried the Enterprise Version with all the extra features. It makes sense if you are a JavaEE developer BUT, like Eclipse, when the IDE activates all these Enteprise Wizards and facets it becomes slow. So I think I can live without them, despite the fact that they might save you some time in a configuration or special annotation. Maybe for less experienced developers these wizard can save you some time, at the time being I can still work with no JavaEE /JSF wizard Java Refactorings : It seems that the tool is more ‘clever’ java way, it spots on the fly common programming errors and provides on the spot suggestions. I have never seen a tool, doing so correct suggestions and scanning. Well done jetbrains team, well done . Searching stuff: Most of the time in fairly large project, finding a class, a resource something is a major repetitve time consuming task. I think that IntelliJ builts on top of the Ecipse legacy, which introduced back in the day fast and smart searching, and does it better. Ohh yes I loved the (Shift+Shift) combo. Quality: As I’ve already said the built in java lang scanning is very good, that means that the tool helps you write better code. The standard ‘Analyze’ functionality provides a variety of suggestions, most of them to the point. I have also installed the PMD, Findbugs, Checkstyle plugins, so I am very happy there is already integration with these very very important tools for ever Java Developer. Text editor:  Smart cursors, each renames and smart support for many different files, things I am not slowly trying to use and explore. App server support: Currently I am using Websphere (bliah) eventually the standard plugin is quite good, I can not fully evaluate it though since Websphere can not run on MacOSX so most of the stuff are just no use for me. Others in the team  though, are successfully using ‘hot swap’ and local deploy with no problem. I guess the tool supports all the major app servers, if it managed to do it properly with Websphere then the others must have been easier. Arquillian + JUnit : This is the one thing that I have not managed to make it work. The JUnit runner in Eclipse was most probaly capable on understanding my configuration and successfuly start Arquillian  with  GlassFish on JUnit tests. At the time being when I try to do the same on IntelliJ I fail miserably, maybe it is missing configuration from my side , dont know, this is the only reason I have eclipse on standy by, sometimes I like to debug while I unit test and currently i can not do it on IntelliJ.So far so good, with some small problems that I can live with though. It seems that our small team at work is slowly migrating over to intelliJ (Community Edition).Reference: Using IntelliJ..for 2 weeks, so far so good from our JCG partner Paris Apostolopoulos at the Papo’s log blog....

Neo4j 2.1: Passing around node ids vs UNWIND

When Neo4j 2.1 is released we’ll have the UNWIND clause which makes working with collections of things easier. In my blog post about creating adjacency matrices we wanted to show how many people were members of the first 5 meetup groups ordered alphabetically and then check how many were members of each of the other groups.           Without the UNWIND clause we’d have to do this: MATCH (g:Group) WITH g ORDER BY g.name LIMIT 5   WITH COLLECT(id(g)) AS groups   MATCH (g1) WHERE id(g1) IN groups MATCH (g2) WHERE id(g2) IN groups   OPTIONAL MATCH path = (g1)<-[:MEMBER_OF]-()-[:MEMBER_OF]->(g2)   RETURN g1.name, g2.name, CASE WHEN path is null THEN 0 ELSE COUNT(path) END AS overlap Here we get the first 5 groups, put their IDs into a collection and then create a cartesian product of groups by doing back to back MATCH’s with a node id lookup. If instead of passing around node ids in ‘groups’ we pass around nodes and then used those in the MATCH step we’d end up doing a full node scan which becomes very slow as the store grows. e.g. this version would be very slow: MATCH (g:Group) WITH g ORDER BY g.name LIMIT 5   WITH COLLECT(g) AS groups   MATCH (g1) WHERE g1 IN groups MATCH (g2) WHERE g2 IN groups   OPTIONAL MATCH path = (g1)<-[:MEMBER_OF]-()-[:MEMBER_OF]->(g2)   RETURN g1.name, g2.name, CASE WHEN path is null THEN 0 ELSE COUNT(path) END AS overlap This is the output from the original query: +-------------------------------------------------------------------------------------------------------------+ | g1.name | g2.name | overlap | +-------------------------------------------------------------------------------------------------------------+ | "Big Data Developers in London" | "Big Data / Data Science / Data Analytics Jobs" | 17 | | "Big Data Jobs in London" | "Big Data London" | 190 | | "Big Data London" | "Big Data Developers in London" | 244 | | "Cassandra London" | "Big Data / Data Science / Data Analytics Jobs" | 16 | | "Big Data Jobs in London" | "Big Data Developers in London" | 52 | | "Cassandra London" | "Cassandra London" | 0 | | "Big Data London" | "Big Data / Data Science / Data Analytics Jobs" | 36 | | "Big Data London" | "Cassandra London" | 422 | | "Big Data Jobs in London" | "Big Data Jobs in London" | 0 | | "Big Data / Data Science / Data Analytics Jobs" | "Big Data / Data Science / Data Analytics Jobs" | 0 | | "Big Data Jobs in London" | "Cassandra London" | 74 | | "Big Data Developers in London" | "Big Data London" | 244 | | "Cassandra London" | "Big Data Jobs in London" | 74 | | "Cassandra London" | "Big Data London" | 422 | | "Big Data / Data Science / Data Analytics Jobs" | "Big Data London" | 36 | | "Big Data Jobs in London" | "Big Data / Data Science / Data Analytics Jobs" | 20 | | "Big Data Developers in London" | "Big Data Jobs in London" | 52 | | "Cassandra London" | "Big Data Developers in London" | 69 | | "Big Data / Data Science / Data Analytics Jobs" | "Big Data Jobs in London" | 20 | | "Big Data Developers in London" | "Big Data Developers in London" | 0 | | "Big Data Developers in London" | "Cassandra London" | 69 | | "Big Data / Data Science / Data Analytics Jobs" | "Big Data Developers in London" | 17 | | "Big Data London" | "Big Data Jobs in London" | 190 | | "Big Data / Data Science / Data Analytics Jobs" | "Cassandra London" | 16 | | "Big Data London" | "Big Data London" | 0 | +-------------------------------------------------------------------------------------------------------------+ 25 rows If we use UNWIND we don’t need to pass around node ids anymore, instead we can collect up the nodes into a collection and then explode them out into a cartesian product: MATCH (g:Group) WITH g ORDER BY g.name LIMIT 5   WITH COLLECT(g) AS groups   UNWIND groups AS g1 UNWIND groups AS g2   OPTIONAL MATCH path = (g1)<-[:MEMBER_OF]-()-[:MEMBER_OF]->(g2)   RETURN g1.name, g2.name, CASE WHEN path is null THEN 0 ELSE COUNT(path) END AS overlap There’s not significantly less code but I think the intent of the query is a bit clearer using UNWIND. I’m looking forward to seeing the innovative uses of UNWIND people come up with once 2.1 is GA.Reference: Neo4j 2.1: Passing around node ids vs UNWIND from our JCG partner Mark Needham at the Mark Needham Blog blog....

Testing effectively

Recently, there was a heaty debate regarding TDD which started by DHH when he claimed that TDD is dead. This ongoing debate managed to capture the attention of developers world, including us. Some mini debates have happened in our office regarding the right practices to do testing. In this article, I will present my own view.   How many kinds of tests have you seen? From the time I joined industry, here are the kinds of tests that I have worked on:Unit Test System/Integration/Functional Test Regression Test Test Harness/Load Test Smoke Test/Spider TestThe above test categories are not necessarily mutually exclusive. For example, you can crate a set of automated functional tests or Smoke tests to be used as regression test. For the benefit of newbie, let’s do a quick review for these old concepts. Unit Test Unit Test aims to test the functionality of a unit of code/component. For Java world, unit of code is the class and each Java class is supposed to have a unit test. The philosophy of Unit Test is simple. When all the components are working, the system as a whole should work. A component rarely works alone. Rather, it normally interacts with other components. Therefore, in order to write Unit Test, developers need to mock other components. This is the problem that DHH and James O Coplien criticize Unit Test for, huge effort that gains little benefit. System/Integration/Functional Test There is no concrete naming as people often use different terms to describe similar things. Contradictory to Unit Test, for functional test, developers aim to test a system function as a whole, which may involve multiple components. Normally, for functional test, the data is retrieved and stored to the test database. Of course, there should be a pre-step to set-up test data before running. DHH likes this kind of test. It helps developers test all the functions of the system without huge effort to set-up mock object. Functional test may involve asserting web output. In the past, it was mostly done with htmlUnit but with the recent improvement of Selenium Grid, Selenium became the preferred choice. Regression Test In this industry, you may end up spending more time maintaining system than developing a new one. Software changes all the time and it is hard to avoid risk whenever you make changes. Regression Test is supposed to capture any defect that caused by changes. In the past, a software house did have one army of testers but the current trend is automated testing. It means that developers will deliver software with a full set of tests that is supposed to be broken whenever a function is spoiled. Whenever a bug is detected, a new test case should be added to cover the new bug. Developers create the test, let it fail, and fix the bug to make it pass. This practice is called Test Driven Development. Test Harness/Load Test Normal test case does not capture system performance. Therefore, we need to develop another set of tests for this purpose. In the simplest form, we can set the time out for the functional test that runs in continuous integration server. The tricky part in this kind of test is that it’s very system dependent and may fail if the system is overloaded. The more popular solution is to run load test manually by using a profiling tool like JMeter or create our own load test app.Smoke Test/Spider Test Smoke Test and Spider Test are two special kinds of tests that may be more relevant to us. WDS provides KAAS (Knowledge as a Service) for the wireless industry. Therefore, our applications are refreshed everyday with data changes rather than business logic changes. It is specific to us that system failure may come from data change rather than business logic. Smoke Tests are set of pre-defined test cases run on integration server with production data. It helps us to find out any potential issues for the daily LIVE deployment. Similar to Smoke Test, Spider Test runs with real data but it works like a crawler that randomly clicks on any link or button available. One of our system contains so many combinations of inputs that it is not possible to be tested by human (closed to 100.000 combinations of inputs). Our Smoke Test randomly chooses some combination of data to test. If it manages to run for a few hours without any defect, we will proceed with our daily/weekly deployment. The Test Culture in our environment To make it short, WDS is a TDD temple. If you create the implementation before writing test cases, better be quiet about it. If you look at WDS self introduction, TDD is mentioned only after Agile and XP “We are:- agile & XP, TDD & pairing, Java & JavaScript, git & continuous deployment, Linux & AWS, Jeans & T-shirts, Tea & cake” Many high level executives in WDS start their career as developers. That helps fostering our culture as an engineering-oriented company. Requesting resources to improve test coverage or infrastructure are common here. We do not have QA. In worst case, Product Owner or customers detect bugs. In best case, we detect bugs by test cases or by team mates during peer review stage. Regarding our Singapore office, most of our team members grew up absorbing Ken Beck and Martin Fowler books and philosophy. That’s why most of them are hardcore TDD worshipers. Even, one member of our team is Martin Fowler’s neighbour. The focus of testing in our working environment did bear fruits. WDS production defects rate is relatively low. My own experience and personal view with testing That is enough about self appraisal. Now, let me share my experience about testing. Generally, Automated Testing works better than QA  Comparing the output of a traditional software house that is packed with an army of QA with a modern Agile team that delivers fully test coverage products, the latter normally outperforms in terms of quality and may even cost effectiveness. Should QA jobs be extinct soon? Over monitoring may hint lack of quality It sounds strange but over the years, I developed an insecure feeling whenever I saw a project that had too many layers of monitoring. Over-monitoring may hint to a lack of confidence and indeed, these systems crash very often with unknown reasons. Writing test cases takes more time that developing features DDH is definitely right on this. Writing Test Cases means that you need to mock input and assert lots of things. Unless you keep writing spaghetti code, developing features take much less time compared to writing tests.UI Testing with javascript is painful You knew it when you did it. Life would be much better if you only needed to test Restful API or static html pages. Unfortunately, the trend of modern web application development involves lots of javascript on client side. For UI Testing, Asynchronous is evil. Whether you want to go with full control testing framework like htmlUnit or using a more practical, generic one like Selenium, it will be a great surprise for me if you never encounter random failures. I guess every developer knows the feeling of failing to get the build pass at the end of the week due to random failure test cases. Developers always over-estimate their software quality It is applicable to me as well because I am an optimistic person. We tend to think that our implementation is perfect until the tests fail or someone helps to point out a bug.Sometimes, we change our code to make writing test cases easier Want it or not, we must agree with DHH on this point. Pertaining to Java world, I have seen people exposing internal variables, creating dummy wrappers for framework objects (like HttpSession, HttpRequest,…) so that it is easier to write Unit Test. DHH finds it so uncomfortable that he chose to walk way from Unit Test. On this part, I half agree and half disagree with him. From my own view, altering design, implementation for the sake of testing is not favourable. It is better if developers can write the code without any concern of mocking input. However, aborting Unit Testing for the sake of having a simple and convenient life is too extreme. The right solution should be designing the system in such a way that business logic won’t be so tight-coupling with framework or infrastructure. This is what is called Domain Driven Design. Domain Driven Design For a newbie, Domain Driven Design gives us a system with the following layers.If you notice, the above diagram has more abstract layers than Rails or the Java adoption of Rails, Play framework. I understand that creating more abstract layers can cause bloated system but for DDD, it is a reasonable compromise. Let’s elaborate further on the content of each layer: Infrastructure This layer is where you store your repository implementation or any other environment specific concerns. For infrastructure, keep the API as simple, dummy as possible and avoid having any business logic implemented here. For this layer, Unit Test is a joke. If there is anything to write, it should be integration test, which works with real database. Domain Domain layer is the most important layer. It contains all system business logic without any framework, infrastructure, environment concern. Your implementation should look like a direct translation of user requirements. Any input, output, parameter are POJO only. Domain layer should be the first layer to be implemented. To fully complete the logic, you may need the interface/API of the infrastructure layer. It is a best practice to keep the API in the Domain Layer and concrete implementation in the Infrastructure layer. The best kind of test cases for the Domain layer is Unit Test as your concern is not the system UI or environment. Therefore, it helps developers to avoid doing dirty works of mocking framework object. For mocking internal state of object, my preferred choice is using a Reflection utility to setup objects rather than exposing internal variables through setters. Application Layer/User Interface Application Layer is where you start thinking about how to represent your business logic to the customer. If the logic is complex or involving many consecutive requests, it is possible to create Facades. Reaching this point, developers should think more about clients than the system. The major concerns should be customer’s devices, UI responsiveness, load balance, stateless or stateful session, Restful API. This is the place for developers to showcase framework talent and knowledge. For this layer, the better kind of test cases is functional/integration test. Similar as above, try your best to avoid having any business logic in Application Layer. Why it is hard to write Unit Test in Rails? Now, if you look back to Rails or Play framework, there is no clear separation of layers like above. The Controllers render inputs, outputs and may contain business logic as well. Similar behaviours applied if you use the ServletAPI without adding any additional layer. The Domain object in Rails is an active record and has a tight-coupling with database schema. Hence, for whatever unit of code that developers want to write test cases, the inputs and output are not POJO. This makes writing Unit Test tough. We should not blame DHH for this design as he follows another philosophy of software development with many benefits like simple design, low development effort and quick feedback. However, I myself do not follow and adopt all of his ideas for developing enterprise applications. Some of his ideas like convention over configuration are great and did cause a major mindset change in developers world but other ideas end up as trade off. Being able to quickly bring up a website may later turn to troubles implementing features that Rails/Play do not support. ConclusionUnit Test is hard to write if your business logic is tight-coupling to framework. Focusing and developing business logic first may help you create better design. Each kinds of components suit different kinds of test cases.This is my own view of Testing. If you have any other opinions, please provide some comments.Reference: Testing effectively from our JCG partner Tony Nguyen at the Developers Corner blog....

Connecting to Cassandra from Java

In my post Hello Cassandra, I looked at downloading the Cassandra NoSQL database and using cqlsh to connect to a Cassandra database. In this post, I look at the basics of connecting to a Cassandra database from a Java client. Although there are several frameworks available for accessing the Cassandra database from Java, I will use the DataStax Java Client JAR in this post. The DataStax Java Driver for Apache Cassandra is available on GitHub. The datastax/java-driver GitHub project page states that it is a “Java client driver for Apache Cassandra” that “works exclusively with the Cassandra Query Language version 3 (CQL3)” and is “licensed under the Apache License, Version 2.0.”   The Java Driver 2.0 for Apache Cassandra page provides a high-level overview and architectural details about the driver. Its Writing Your First Client section provides code listings and explanations regarding connecting to Cassandra with the Java driver and executing CQL statements from Java code. The code listings in this post are adaptations of those examples applied to my example cases. The Cassandra Java Driver has several dependencies. The Java Driver 2.0 for Apache Cassandra documentation includes a page called Setting up your Java development environment that outlines the Java Driver 2.0′s dependencies: cassandra-driver-core-2.0.1.jar (datastax/java-driver 2.0), netty-3.9.0-Final.jar (netty direct), guava-16.0.1.jar (Guava 16 direct), metrics-core-3.0.2.jar (Metrics Core), and slf4j-api-1.7.5.jar (slf4j direct). I also found that I needed to place LZ4Factory.java and snappy-java on the classpath. The next code listing is of a simple class called CassandraConnector. CassandraConnector.java package com.marxmart.persistence;import com.datastax.driver.core.Cluster; import com.datastax.driver.core.Host; import com.datastax.driver.core.Metadata; import com.datastax.driver.core.Session;import static java.lang.System.out;/** * Class used for connecting to Cassandra database. */ public class CassandraConnector { /** Cassandra Cluster. */ private Cluster cluster;/** Cassandra Session. */ private Session session;/** * Connect to Cassandra Cluster specified by provided node IP * address and port number. * * @param node Cluster node IP address. * @param port Port of cluster host. */ public void connect(final String node, final int port) { this.cluster = Cluster.builder().addContactPoint(node).withPort(port).build(); final Metadata metadata = cluster.getMetadata(); out.printf("Connected to cluster: %s\n", metadata.getClusterName()); for (final Host host : metadata.getAllHosts()) { out.printf("Datacenter: %s; Host: %s; Rack: %s\n", host.getDatacenter(), host.getAddress(), host.getRack()); } session = cluster.connect(); }/** * Provide my Session. * * @return My session. */ public Session getSession() { return this.session; }/** Close cluster. */ public void close() { cluster.close(); } } The above connecting class could be invoked as shown in the next code listing. Code Using CassandraConnector /** * Main function for demonstrating connecting to Cassandra with host and port. * * @param args Command-line arguments; first argument, if provided, is the * host and second argument, if provided, is the port. */ public static void main(final String[] args) { final CassandraConnector client = new CassandraConnector(); final String ipAddress = args.length > 0 ? args[0] : "localhost"; final int port = args.length > 1 ? Integer.parseInt(args[1]) : 9042; out.println("Connecting to IP Address " + ipAddress + ":" + port + "..."); client.connect(ipAddress, port); client.close(); } The example code in that last code listing specified default node and port of localhost and port 9042. This port number is specified in the cassandra.yaml file located in the apache-cassandra/conf directory. The Cassandra 1.2 documentation has a page on The cassandra.yaml configuration file which describes the cassandra.yaml file as “the main configuration file for Cassandra.” Incidentally, another important configuration file in that same directory is cassandra-env.sh, which defines numerous JVM options for the Java-based Cassandra database. For the examples in this post, I will be using a MOVIES table created with the following Cassandra Query Language (CQL): createMovie.cql CREATE TABLE movies ( title varchar, year int, description varchar, mmpa_rating varchar, dustin_rating varchar, PRIMARY KEY (title, year) ); The above file can be executed within cqlsh with the command source 'C:\cassandra\cql\examples\createMovie.cql' (assuming that the file is placed in the specified directory, of course) and this is demonstrated in the next screen snapshot.One thing worth highlighting here is that the columns that were created as varchar datatypes are described as text datatypes by the cqlsh describe command. Although I created this table directly via cqlsh, I also could have created the table in Java as shown in the next code listing and associated screen snapshot that follows the code listing. Creating Cassandra Table with Java Driver final String createMovieCql = "CREATE TABLE movies_keyspace.movies (title varchar, year int, description varchar, " + "mmpa_rating varchar, dustin_rating varchar, PRIMARY KEY (title, year))"; client.getSession().execute(createMovieCql); The above code accesses an instance variable client. The class with this instance variable that it might exist in is shown next. Shell of MoviePersistence.java package dustin.examples.cassandra;import com.datastax.driver.core.ResultSet; import com.datastax.driver.core.Row;import java.util.Optional;import static java.lang.System.out;/** * Handles movie persistence access. */ public class MoviePersistence { private final CassandraConnector client = new CassandraConnector();public MoviePersistence(final String newHost, final int newPort) { out.println("Connecting to IP Address " + newHost + ":" + newPort + "..."); client.connect(newHost, newPort); }/** * Close my underlying Cassandra connection. */ private void close() { client.close(); } } With the MOVIES table created as shown above (either by cqlsh or with Java client code), the next steps are to manipulate data related to this table. The next code listing shows a method that could be used to write new rows to the MOVIES table. /** * Persist provided movie information. * * @param title Title of movie to be persisted. * @param year Year of movie to be persisted. * @param description Description of movie to be persisted. * @param mmpaRating MMPA rating. * @param dustinRating Dustin's rating. */ public void persistMovie( final String title, final int year, final String description, final String mmpaRating, final String dustinRating) { client.getSession().execute( "INSERT INTO movies_keyspace.movies (title, year, description, mmpa_rating, dustin_rating) VALUES (?, ?, ?, ?, ?)", title, year, description, mmpaRating, dustinRating); } With the data inserted into the MOVIES table, we need to be able to query it. The next code listing shows one potential implementation for querying a movie by title and year. Querying with Cassandra Java Driver /** * Returns movie matching provided title and year. * * @param title Title of desired movie. * @param year Year of desired movie. * @return Desired movie if match is found; Optional.empty() if no match is found. */ public Optional<Movie> queryMovieByTitleAndYear(final String title, final int year) { final ResultSet movieResults = client.getSession().execute( "SELECT * from movies_keyspace.movies WHERE title = ? AND year = ?", title, year); final Row movieRow = movieResults.one(); final Optional<Movie> movie = movieRow != null ? Optional.of(new Movie( movieRow.getString("title"), movieRow.getInt("year"), movieRow.getString("description"), movieRow.getString("mmpa_rating"), movieRow.getString("dustin_rating"))) : Optional.empty(); return movie; } If we need to delete data already stored in the Cassandra database, this is easily accomplished as shown in the next code listing. Deleting with Cassandra Java Driver /** * Deletes the movie with the provided title and release year. * * @param title Title of movie to be deleted. * @param year Year of release of movie to be deleted. */ public void deleteMovieWithTitleAndYear(final String title, final int year) { final String deleteString = "DELETE FROM movies_keyspace.movies WHERE title = ? and year = ?"; client.getSession().execute(deleteString, title, year); } As the examples in this blog post have shown, it’s easy to access Cassandra from Java applications using the Java Driver. It is worth noting that Cassandra is written in Java. The advantage of this for Java developers is that many of Cassandra’s configuration values are JVM options that Java developers are already familiar with. The cassandra-env.sh file in the Cassandra conf directory allows one to specify standard JVM options used by Cassandra (such as heap sizing parameters -Xms, -Xmx, and -Xmn),HotSpot-specific JVM options (such as -XX:-HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath, garbage collection tuning options, and garbage collection logging options), enabling assertions (-ea), and exposing Cassandra for remote JMX management. Speaking of Cassandra and JMX, Cassandra can be monitored via JMX as discussed in the “Monitoring using JConsole” section of Monitoring a Cassandra cluster. The book excerpt The Basics of Monitoring Cassandra also discusses using JMX to monitor Cassandra. Because Java developers are more likely to be familiar with JMX clients such as JConsole and VisualVM, this is an intuitive approach to monitoring Cassandra for Java developers. Another advantage of Cassandra’s Java roots is that Java classes used by Cassandra can be extended and Cassandra can be customized via Java. For example, custom data types can be implemented by extending the AbstractType class. Conclusion The Cassandra Java Driver makes it easy to access Cassandra from Java applications. Cassandra also features significant Java-based configuration and monitoring and can even be customized with Java.Reference: Connecting to Cassandra from Java from our JCG partner Dustin Marx at the Inspired by Actual Events blog....

Is it Imperative that you learn Functional Programming with Java 8?

I’ve recently been taking look at Java 8 and have got hold of “Java 8 In Action” published by Manning. The first thing that struck me is how one of Java 8′s unique sales propositions is functional programming; functions are now first class variables, you can pass them around your code as you would an int or a String. This is a big change. It seems that functional languages have become more popular in recent years and there are no end of them to choose from. Modern function programming language examples include Clojure, JavaScript, Scala, and even Erlang, invented in the late 1980s, has made a come back. So, why is there this change in direction? You could probably come up with several reasons, but we’ll begin with the premise that industry best practise changes over time and even the most popular languages will one day fall out of favour. I imagine that if you’re young enough there will be a day when you’ll look back and say “remember when we used to use Java”? Before looking at why there is this change, let’s reflect on how we got here by stepping back to the 1980s…   According to Wikipedia the IBM PC was released on 12th August 1981. The first PCs shipped with BASIC 1 although the cool guys soon got fed up with this and moved on to Borland’s Turbo Pascal. Turbo Pascal’s coolness didn’t last that long because Microsoft bought out Microsoft C (or MSC), which soon became the defacto standard. It was cool because if you were clever you could access the hardware directly using int 21H, int 10H and int 14h and others, and if can remember which interrupt did what, then you’re old like me 2… There were other computers before the IBM PC including the Apple II, Commodore Pet etc, but the IBM PC was a ‘grown-up’ PC for business applications. I remember trying to buy one for a project I did in 1985, they cost thousands of pounds and you couldn’t get one for love nor money, everyone wanted one; however, I digress.In the late 1980s came the Microsoft Windows SDK, a C based SDK that’s bean around since tiled Windows 1 (apparently Microsoft didn’t used overlapping windows in Windows 1 because it copied Apple’s Macintosh and probably infringed their patents – even though Apple allegedly stole the idea for the Mac from Xerox Parc, though how true this is I can’t confirm). The SDK really took off in Windows 2 introducing the world to callback, message based programming; an idea they allegedly stole from X-Windows and Motif on Unix.   Around the mid 1990s languages became Object Oriented; classes were invented to tie together methods and data, introducing the concepts of data hiding and encapsulation. ‘C’ became ‘C++’ and if you were using the Windows SDK around this time then you switched to Microsoft Foundation Classes; an OO wrapper around the SDK. This change was seen as a good thing. A major problem with the old function based languages was that you could allocate global variables anywhere in your code base and change them using any function at any time. This obviously caused major screw-ups in a lot systems as you couldn’t be sure that changing the state of a global variable wouldn’t introduce a bug in some far flung corner of your application. The other major problem with defacto language C was that YOU were responsible for allocating and deallocating memory and if memory pointers were global then, when you accessed them, you couldn’t be 100% certain that the pointer was still valid and if it wasn’t… you crashed. Along with object oriented languages came object oriented methodologies, culminating in the late 1990s with UML. This was a fusion of the Booch Method, James Rumbaugh’s OMT and Ivor Jacobsen’s OOSE and it was de rigueur when it came to designing software. All kinds of tools were available for the job of documenting and communicating your designs and, from experience, some of these were of very questionable quality, which in begs the question: were the developers actually using UML to write their UML programs? If so, then those products weren’t really a good advert for UML. You don’t see as many organisations using UML these days, though I still do when I need to get a design straight in my head. My first choice of UML tool is, and always will be, pencil and paper. It’s simple and it works. Finally in my brief history of programming is Java. Initially released in 1995 and becoming popular a few years later, Java was based upon the idea of improved C++. This mainly because it runs on its own virtual machine, which takes care of memory allocation and deallocation for you. It has become a defacto standard of object orientated languages. The thing about this broadly made up timeline is the under pinning concept of imperative programming. In summary, assembly language gave rise to C, C to C++ and OO, and C++ and OO lead to Java – all imperative. Wikipedia gives a good overview of imperative programming, so I’m not going into it in detail, but I’ll summarise imperative programming as programming with functions AND mutable state, meaning you can have instance and global variables. How does functional programming differ to imperative programming? The main idea is that functions are data, just like integers and Strings; algorithms are implemented in terms of function calls (while and for loops do not exist, you use recursion) and variables are always local. You could be fooled in to thinking that, because they contain the concept of a function and no classes, languages such as C and Turbo Pascal are functional programming languages. They’re not, they’re imperative programming languages, because they have state data. So, what’s changed? The common answer to this, one that put about by Erlang developers and one you’ll find in Manning’s Java 8 book, is that hardware has changed. ‘Computers’, at least the one’s you’ll find in a server room, are now a sophisticated multi-processor, multi-core affairs with terabytes of memory. Take, for example, the HP Proliant DL580 G8 server; it has up to four processors and each processor can have up to 15 64bit cores. That’s huge, especially when compared with the original, revolutionary 16bit Intel 8086 on the original IBM PC. Assuming you’re running Unix and you ran the top command, then maximum processor usage would be given as 6000%. Given this dramatic rise in machine power we software developers need languages that can support them allowing us to easily use all this concurrent processing power and that’s were functional programming comes in. In imperative programming you can have instance variables, global variables and mutable state. These can be shared between threads although sharing these resources is costly and inefficient in terms of synchronisation and locking. It’s also fairly slow and difficult to do as you have to avoid deadlocks and other similar problems. Functional programming does away with all these problems because it does away with all the tedious mucking about with instance variables and state. That means you don’t have to bother with locking and synchronisation and thread or processes (call them what you will) can be independent of each other. That’s the theory, but does it hold up to scrutinisation? Let’s not forget that it’s possible to write good multi-threaded programs that effectively use the many cores of a large multi-core and multi-processor machine with imperative Java 6 or 7. Just like functional programming languages you have to think about what you’re doing, plan out a suitable design and execute it using business ‘best practice’. Just because Java 6 or 7 is an imperative language, you don’t have to share data between threads/processes and use synchronisation and locking, it’s just a matter of design. The logical conclusion of this is that you can do without functional programming, which leads to possibly the real reason behind the popularity of functional programming languages: people enjoy using them. You could therefore say that functional languages are the “latest thing”; a craze, they’re trendy, a fashion. I have to emphasise that ‘fashion’ in software development is not necessarily a bad thing and is nothing new. If you go back to my potted history above you’ll see that the time-line is full of trends and crazes: the adoption of ‘C’ over Turbo Pascal and BASIC, the paradigm shift to Object Orientation and even the move to Java with it’s compile once and run anywhere promise. Is it Imperative that you learn Functional Programming with Java 8? Ask me in a few years time…   1All historical facts in this blog are guaranteed inaccurate by my failing memory. 2Correct me if I’m wrong, but int 21H = MSDOS Functions, int 10H = direct screen access and int 14H = serial I/OReference: Is it Imperative that you learn Functional Programming with Java 8? from our JCG partner Roger Hughes at the Captain Debug’s Blog blog....

SharedHashMap vs Redis

Overview This is a comparison between OpenHFT’s SharedHashMap and a popular key-value store Redis. Any vendor will tell you how great their product is, so I will start by outlining why you wouldn’t use SharedHashMap, before I tell you why it is a “must have” for performant applications.       Why you would use Redis? Redis is a more mature database, relatively widely used and it includes;Support for multiple languages. Access over TCP to remote clients. A command line management tool. It out performs many other key-value stores.Why you would use OpenHFT’s SharedHashMap? You need to maximise performance in Java. It outperforms Redis, and many other popular key-value stores by more than an order of magnitude in Java. Why does SharedHashMap out perform Redis? It is designed for performance from the start, by being as lightweight as possible.It acts as an embedded data store, even across multiple processes. You don’t pay the price of TCP messaging via the kernel. It was designed to be used in Java in a pause less, garbage free manner. It is written in Java, for Java.But C is faster than Java?It is only faster when you compare like for like, and even then, not always.  However if you compare an embedded data store written in Java to one which must pass over TCP and translate between languages, the embedded data store is much faster. How much difference does it make?Benchmarks can be bend to suit any argument.  Vendor benchmarks tend to give you the most optimistic numbers because they know what their product will do best.  The simplest benchmarks for key-value stores are the same, and one of them is to start with an empty data store and setting lots of small key-values, and that at least gives you an ideal of the best you can hope for.  Your use case is likely to be slower, with more complex requirements. Setting millions of key-values on a 16 core server with 128 GB of memory. Redis   SharedHashMapSingle threaded      ~10K updates/sec     ~3M updates/secMulti-threaded    ~100K updates/sec   ~30M updates/sec  The numbers are approximate, but they were performed on the same machine for the same operations in Java (using Jedis to connect to Redis)  YMMWV Conclusion Accurate and impartial benchmarks are a myth, but they do accurately serve you in giving a vendors view of a product.  OpenHFT’s view of SharedHashMap is that it is designed for Java and is performed in a way that many popular key-values stores cannot match.  If you need to maximise the efficiency of your Java system, you should be considering OpenHFT’s data management products.Reference: SharedHashMap vs Redis from our JCG partner Peter Lawrey at the Vanilla Java blog....

Scaling Agile? Think Out, Not Up

I taught the Influential Agile Leader workshop with Gil Broza last week in Edinburgh. (That’s why I was quiet. I was traveling and teaching. No time for writing, blogging or tweeting.) One of the participants asked me what I thought about scaling agile. Before I could explain about small-world networks, not hierarchies, he said, “I am sure the way you scale agile is out, not up. Well, blow me over with a feather. He said it more simply than I did. If you look at my picture of a technical program team, you can see that’s how it works.  The technical program team has feature teams alone, if they can be alone. Joe, Tim, and Henry all have stand-alone feature teams. If they need to be “collected” because they work on related features, they collect themselves. Sally has collected feature teams. The teams scale out, at the technical level, not up. The technical program team does not have to get to bigger. When I ran programs in the past, I emailed the program team meeting agenda (it was a problem solving meeting) to everyone, and say, “Here are the people I need to attend. Everyone else: let me know if you are attending.”   Now, there’s a limit to how big a program can get for the software program or the hardware program. At some point, the it’s so hard to coordinate the interdependencies, it’s not worth the bigness. If the teams are delivering small features all the time, you don’t need as many people as you think you do. The smaller the batch size, the fewer the people required. Your momentum will be greater. If you don’t believe me, think about that for a minute or two. When you think scaling agile, think out, not up. You use small world networks, and when you say, “think out, not up,” it’s a very nice catch-phrase.Reference: Scaling Agile? Think Out, Not Up from our JCG partner Johanna Rothman at the Managing Product Development blog....

My Move From ColdFusion to Java Development

Although I studied and experimented with different development technologies and tools throughout my college and graduate careers, my full-time professional career began with development in Adobe ColdFusion. Coming out of school, solving real-world business problems using technology was a challenge in itself. Due to its fairly uncomplicated nature, ColdFusion did not stand in the way of getting things done. It allowed me to apply the main principles of application development, too. Over time, however, I began to notice that the tool set offered by the language was not as rich as that of other development technologies like Java, for example.   Eventually I developed a desire to work more closely with a more complex but rewarding language, and that is when I transitioned to Java. I often think of this transition as going back to my roots, since my first meaningful experiences developing applications were in Java. This blog will discuss some of my thoughts and experiences in making the transition. ColdFusion – making rudimentary tasks easy In a typical web application stack, ColdFusion is the server-side development technology that fulfills requests received from a user’s web browser and sends meaningful information back. ColdFusion is a tag-based language that integrates with HTML very well. It can also stand on its own in an object-oriented manner, but that doesn’t change its purpose. While I was still working with ColdFusion, my experiences were providing me with piecemeal hints about the sort of resources Java has in store. Because ColdFusion itself is a higher level interpreted language that runs on top of Java, it was possible to use the Java underneath in creative ways. For example, the following code snippet allows to remove duplicates from a ColdFusion array in a single statement1. myArray = createObject("java", "java.util.ArrayList").init( createObject("java", "java.util.HashSet").init(myArray) ); These techniques, however, were by and large undocumented. You would already have to know some neat tricks in Java in order to apply them in ColdFusion. And if that’s the case, why not invest time in getting better at Java? I believe that ColdFusion has progressed considerably over time, but only to uncover and implement what Java already mastered. While the underpinnings of the two technologies are very similar, ColdFusion has a specific purpose on the market. It prides itself on making rudimentary tasks easy. For example, database access facilities in ColdFusion are very laconic, especially for the time when they were first introduced. By virtue of this sort of ease of use, ColdFusion has created a reputation of being a Rapid Application Development platform. Not only are the server-side facilities convenient to use, the language offers some out-of-the-box UI components that can potentially save development time and, arguably, relieve the developer of front end duties to some degree. It sounds good in theory. In practice, however, the UI widgets are too rudimentary to be used in any modern real world application, especially considering the front-end libraries that have emerged in recent years, not to mention the emerging capabilities of HTML5. The built-in shortcuts for making AJAX calls do seem elegant, but they are often not flexible enough for all scenarios, so you would end up resorting to more robust jQuery anyway. When you only take the good parts, ColdFusion is “just another” server-side technology that also happens to bring an organizational culture along with it, or it creates an organizational culture that, in my opinion, is not ideal for the long term. Rapidity The notion of Rapid Application Development is often misconstrued. In fact, the rapidity of it doesn’t seem to buy you much in the end. Especially when first setting up an application, a fair amount of thought and planning should be given to the architecture. If the proper architecture is in place, making enhancements to the application will not be terribly tasking. On the other hand, if something is developed “rapidly” and hastily, the inevitable technical debt will be forever weighing you down, as it is much harder to justify spending the time to refactor an application than to create an enhancement. Often times, refactoring takes longer, introduces system-wide bugs, which requires additional QA resources. The more that I continue with this thought, the more I realize how important it is to get the foundation right and the word “rapid” doesn’t seem appealing in this context. With that in mind, I have experienced different performance expectations in workplaces where these two technologies are used. As you might have guessed, the timeframes for completing functionality have been consistently more demanding where ColdFusion is used. I am definitely a proponent of productive working environments, but I also believe there should be a balance between delivering functionality and maintaining the code base in such a way that future enhancements and fixes can be completed more easily. It is difficult to maintain a culture of quality when the focus is on the application features alone. I have found that environments where more sensible architecture is used allow for some “buffer time” to leave the code base a better place than when you encountered it, much like the Boy Scout rule. Tooling Another point worth touching on is the level of sophistication and usefulness of the development tools. Both for my ColdFusion and Java development work I utilized Eclipse as the integrated development environment. Eclipse is traditionally known as a Java IDE. However, due to its extensibility, ColdFusion development can be facilitated via a plug-in. While the community support for this plug-in is excellent, it is no match for the code inspection and debugging tools available for Java. I tend to think that the difference is due to the fact that Java is a compiled language, whereas ColdFusion is interpreted at runtime. Whatever the case, considering that Eclipse is a free resource, it improves developer productivity tremendously. It was a rediscovered luxury when I started working with Java in my professional career. I became convinced that it is not necessarily the development technology itself that can save you time, but the tools you use throughout the development process. The access to useful tools available at no cost certainly makes Java a leader in this comparison. Final Thoughts Whether incidental or not, different technologies seem to foster organizational culture to a certain extent and have their own strengths and weaknesses. Ultimately, no matter what you prefer to develop in, it is important to continue to grow as a professional and to challenge yourself to find new and creative ways to use a technology or development language. I have found the Java platform to be very rewarding in this regard. Learning something new everyday might seem overwhelming, but, just like in the case of performing physical exercises, the mind becomes more effective over time. Resources“Use ColdFusion? Use Java.” by Barney BoisvertReference: My Move From ColdFusion to Java Development from our JCG partner Dmitry Kolesnikov at the Keyhole Software blog....

Create a jar library with gradle using AAR info

Some posts ago, I talked about how to use gradle to push an aar to maven central. If you remember, we had to modify some files and so on, but the work we have to do helps other developers to simplify their development when they want to use our code/library. When our code is pushed to the maven central as aar we can resuse it as libray simply setting a gradle dependency. For example, if we want to use Weatherlib we have to write:         dependencies { compile fileTree(dir: 'libs', include: ['*.jar']) compile 'com.android.support:appcompat-v7:19.+' compile 'com.survivingwithandroid:weatherlib:1.3.1' } Very simple! Anyway, this is true if we use Android Studio, but what if we use Eclipse or something else? In some cases it is easier to have a classic jar that can be imported in our Eclipse project and add a dependency in our project to this jar. If we use Android Studio this jar isn’t created easily (AFAIK) and I had to slightly modify the build.gradle to create the jar. I looked around on the net and I found a solution that I re-adapted so that we can reuse the information stored in the properties file. If you remember from the post about aar and gradle (if not look here), there are two properties files that I show it for simplicity: POM_NAME=Android Weather Library POM_ARTIFACT_ID=weatherlib POM_PACKAGING=aar and VERSION_NAME=1.3.1 VERSION_CODE=6 GROUP=com.survivingwithandroidPOM_DESCRIPTION=Android Weather Lib POM_URL=https://github.com/survivingwithandroid/WeatherLib POM_SCM_URL=https://github.com/survivingwithandroid/WeatherLib POM_SCM_CONNECTION=scm:git@github.com:survivingwithandroid/weatherlib.git POM_SCM_DEV_CONNECTION=scm:git@github.com:survivingwithandroid/weatherlib.git POM_LICENCE_NAME=The Apache Software License, Version 2.0 POM_LICENCE_URL=http://www.apache.org/licenses/LICENSE-2.0.txt POM_LICENCE_DIST=repo POM_DEVELOPER_ID=survivingwithandroid POM_DEVELOPER_NAME=Francesco Azzola So I would like to use this information to create a jar with the name equals to the POM_ARTIFACT_ID combined with the VERSION_NAME, and this jar should be created under a specific directory. So we have to add under android section in build.gradle: android { ...sourceSets { main { java { srcDir 'src/main/java' } resources { srcDir 'src/../lib' } } } .. } and after the dependencies section: task clearJar(type: Delete) { delete 'build/libs/' + POM_ARTIFACT_ID + '_' + VERSION_NAME + '.jar' }task makeJar(type: Copy) { from('build/bundles/release/') into('release/') include('classes.jar') rename ('classes.jar', POM_ARTIFACT_ID + '_' + VERSION_NAME + '.jar') }makeJar.dependsOn(clearJar, build) Now if you run the task makeJar, AS will create a jar under the directory called release.If you want to have the build.gradle file you can get it here.Reference: Create a jar library with gradle using AAR info from our JCG partner Francesco Azzola at the Surviving w/ Android blog....

Playing with Java 8 – Lambdas, Paths and Files

I needed to read a whole bunch of files recently and instead of just grabbing my old FileUtils.java that I and probably most developers have and then copy from project to project, I decided to have quick look at how else to do it… Yes, I know there is Commons IO and Google IO, why would I even bother?  They probably do it better, but I wanted to check out the NIO jdk classes and play with lambdas as well…and to be honest, I think this actually ended up being a very neat bit of code. So I had a specific use case: I wanted to read all the source files from a whole directory tree, line by line. What this code does, it uses Files.walk to recursively get all the paths from the starting point, it creates a stream, which I then filter to only files that end with the required extension. For each of those files, I use Files.lines to create a stream of Strings, one per line. I trim that, filter out the empty ones and add them to the return collection. All very concise thanks to the new constructs. package net.briandupreez.blog.java8.io;import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory;import java.io.IOException; import java.nio.charset.Charset; import java.nio.file.FileVisitOption; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.util.ArrayList; import java.util.List; import java.util.stream.Stream;/** * RecursiveFileLineReader * Created by Brian on 2014-05-26. */ public class RecursiveFileLineReader {private transient static final Log LOG = LogFactory.getLog(RecursiveFileLineReader.class);/** * Get all the non empty lines from all the files with the specific extension, recursively. * * @param path the path to start recursion * @param extension the file extension * @return list of lines */ public static List<String> readAllLineFromAllFilesRecursively(final String path, final String extension) { final List<String> lines = new ArrayList<>(); try (final Stream<Path> pathStream = Files.walk(Paths.get(path), FileVisitOption.FOLLOW_LINKS)) { pathStream .filter((p) -> !p.toFile().isDirectory() && p.toFile().getAbsolutePath().endsWith(extension)) .forEach(p -> fileLinesToList(p, lines)); } catch (final IOException e) { LOG.error(e.getMessage(), e); } return lines; }private static void fileLinesToList(final Path file, final List<String> lines) { try (Stream<String> stream = Files.lines(file, Charset.defaultCharset())) { stream .map(String::trim) .filter(s -> !s.isEmpty()) .forEach(lines::add); } catch (final IOException e) { LOG.error(e.getMessage(), e); } }}Reference: Playing with Java 8 – Lambdas, Paths and Files from our JCG partner Brian Du Preez at the Zen in the art of IT blog....
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books