Featured FREE Whitepapers

What's New Here?


Using Markdown syntax in Javadoc comments

In this post we will see how we can write Javadoc comments using Markdown instead of the typical Javadoc syntax. So what is Markdown?Markdown is a plain text formatting syntax designed so that it optionally can be converted to HTML using a tool by the same name. Markdown is popularly used to format readme files, for writing messages in online discussion forums or in text editors for the quick creation of rich text documents. (Wikipedia: Markdown)Markdown is a very easy to read formatting syntax. Different variations of Markdown can be used on Stack Overflow or GitHub to format user generated content. Setup By default the Javadoc tool uses Javadoc comments to generate API documentation in HTML form. This process can be customized used Doclets. Doclets are Java programs that specify the content and format of the output of the Javadoc tool. The markdown-doclet is a replacement for the standard Java Doclet which gives developers the option to use Markdown syntax in their Javadoc comments. We can set up this doclet in Maven using the maven-javadoc-plugin. <build>   <plugins>     <plugin>       <artifactId>maven-javadoc-plugin</artifactId>       <version>2.9</version>       <configuration>         <doclet>ch.raffael.doclets.pegdown.PegdownDoclet</doclet>         <docletArtifact>           <groupId>ch.raffael.pegdown-doclet</groupId>           <artifactId>pegdown-doclet</artifactId>           <version>1.1</version>         </docletArtifact>         <useStandardDocletOptions>true</useStandardDocletOptions>       </configuration>     </plugin>   </plugins> </build> Writing comments in Markdown Now we can use Markdown syntax in Javadoc comments: /**  * ## Large headline  * ### Smaller headline  *  * This is a comment that contains `code` parts.  *  * Code blocks:  *  * ```java  * int foo = 42;  * System.out.println(foo);  * ```  *  * Quote blocks:  *  * > This is a block quote  *  * lists:  *  *  - first item  *  - second item  *  - third item  *  * This is a text that contains an [external link][link].  *  * [link]: http://external-link.com/  *  * @param id the user id  * @return the user object with the passed `id` or `null` if no user with this `id` is found  */ public User findUser(long id) {   ... } After running mvn javadoc:javadoc we can find the generated HTML API documentation in target/site/apidocs. The generated documentation for the method shown above looks like this:As we can see the Javadoc comments get nicely converted to HTML. Conclusion Markdown has the clear advantage over standard Javadoc syntax that the source it is far easier to read. Just have a look at some of the method comments of java.util.Map. Many Javadoc comments are full with formatting tags and are barely readable without any tool. But be aware that Markdown can cause problems with tools and IDEs that expect standard Javadoc syntax.Reference: Using Markdown syntax in Javadoc comments from our JCG partner Michael Scharhag at the mscharhag, Programming and Stuff blog....

NoSQL – A Quick Guide

NoSQL is buzz word nowadays among the developers and software professionals. 1. What is NoSQL ? NoSQL database, also called Not Only SQL, is an approach to data management and database design that’s useful for very large sets of distributed data. 2. Where to use NoSQL ? Use NOSQL, When project has unstructured big data that require real-time or offline analysis or web/mobile application. i.e. Social Network app, Analytics app. 3. Advantages and Disadvantages of NoSQL DB Advantages of NoSQLElastic scaling Big Data Ecomomics Flexible data modelsDisadvantages of NoSQLMaturity Support Analytics and business intelligence Administration Expertise4. Category of NoSQLColumn Document Key-value Graph5. How many NoSQL database are available in market ? More than 110 different (Open Source and Proprietary) NoSQL database available in market. 6. If all NoSQL database fall under above category then what is purpose of having lots of NoSQL databases ? Every NOSQL database has some special feature & functionality which makes it different, Base on the project requirement one can choose NOSQL database. 7. Can I use multiple NoSQL in my project / application ? Yes. 8. List of popular NoSQL database with usage Radis: For rapidly changing data (should fit mostly in memory). i.e. to store real-time stock prices, analytics, leaderboards and communication. And replacement of memcached. MongoDB: When you need dynamic queries, defined indexes, map/reduce and good performance on a big DB. i.e. for most things that you would do with MySQL but having predefined columns really holds you back. Cassandra: When you need to store data so huge that it doesn’t fit on server, but still want a friendly familiar interface to it. When you don’t need real-time analysis or other operation. i.e. Web analytics, Transaction logging, Data collection from huge sensor arrays. Riak: If you need very good single-site scalability, availability and fault-tolerance, but you’re ready to pay for multi-site replication. i.e. Point-of-sales data collection. Factory control systems. Places where even seconds of downtime hurt. Could be used as a well-update-able web server. CouchDB: For accumulating, occasionally changing data, on which pre-defined queries are to be run. Places where versioning is important. i.e. CRM, CMS systems. Master-master replication is an especially interesting feature, allowing easy multi-site deployments. HBase: Hadoop is probably still the best way to run Map/Reduce jobs on huge datasets. Best if you use the Hadoop/HDFS stack already. ie. Search engines. Analysing log data. Any place where scanning huge, two-dimensional join-less tables are a requirement. Accumulo: If you need to restict access on the cell level. i.e. Same as HBase, since it’s basically a replacement: Search engines. Hypertable: If you need a better HBase. i.e/ Same as HBase, since it’s basically a replacement: Search engines. Neo4j: For graph-style, rich or complex, interconnected data. Neo4j is quite different from the others in this sense. i.e. For searching routes in social relations, public transport links, road maps, or network topologies. ElasticSearch: When you have objects with (flexible) fields, and you need “advanced search” functionality. i.e. A dating service that handles age difference, geographic location, tastes and dislikes, etc. Or a leaderboard system that depends on many variables. you can replace your Solr with ElasticSearch. Couchbase: Any application where low-latency data access, high concurrency support and high availability is a requirement. i.e. Low-latency use-cases like ad targeting or highly-concurrent web apps like online gaming (e.g. Zynga).Reference: NoSQL – A Quick Guide from our JCG partner Ketan Parmar at the KP Bird blog....

So You Want to Use A Recruiter Part I – Recruit Your Recruiter

This is the first in a three-part series to inform job seekers about working with a recruiter. Part II is “Establishing Boundaries” and Part III is “Warnings” This week I read an unusually high number of articles (and the comments!) about recruiting. Although most of the discussion quickly turns to harsh criticism, there are always a few people wondering the best ways to find a decent recruiter to work with and what to do once they have established contact. Some recruiter demand stems from candidates looking to relocate into areas where they have no network, while others just want to maximize their options and feel they may benefit from the services provided by an agency recruiter. Regardless of your reasons for seeking out an agency recruiter, first you have to find one. Finding a Recruiter There are three reliable methods to getting introduced to a recruiter. Referral This is the best method for most, as being introduced by a contact can have unexpected benefits. To maximize those benefits, you must consider the source of your referral. If the recruiter has a great deal of respect for the person introducing you, you are likely to be given some immediate credibility and favorable treatment due to that association. Unfortunately the alternative is true, and if you are referred by someone the recruiter does not respect it may be assumed that you are not a strong talent. When asking for recruiter referrals it is wise to start with the most talented people in your network. Your network does not have to be the only source of referrals, particularly if you are looking for a recruiter in an area where you have no network. User group and meetup leaders are frequently contacted by recruiters and one should expect group leaders to be knowledgeable of the local market. Even a random email to an engineer in another location could result in a solid lead. Let the recruiter find you If I had a nickel for every time I heard technologists complain that they aren’t hearing from enough recruiters, I’d be poor – though some voice frustration that they don’t hear from the ‘right ones‘. Increasing your visibility will attract recruiters who may or may not be the ones you’d want, but it helps establish a pool for evaluation to choose who is worthy of a response. To maximize your chances of being found and contacted, you need to consider how recruiters will find you. The obvious place is LinkedIn, and spending a few minutes fixing up your profile will help. Keywords and SEO concepts as well as profile ‘completeness’ should be your focus. (further reading on this) Recruiters are likely to be searching for combinations of keywords from their requirements, usually with some advanced search filters based on location, education, or experience. Completeness matters. Some recruiters search Twitter and the other standard social sites as well. If you have a profile anywhere, just assume that a recruiter might find it and optimize keywords similarly. Keep in mind how easy or difficult it is for people to contact you once you’ve been found. Just because I see your LinkedIn profile or Google + account doesn’t mean I can contact you. Many professionals create an email address (maybe currentemail-jobs@domain) strictly for recruiter correspondence and include it in their LinkedIn profile and other social pages. Another option is to get discovered on job search sites like Indeed, Monster, and Dice. These are frequented by active job seekers, and some recruiters may view your posting there as a somewhat negative signal. Be warned that posting personal information on these sites means that those phone numbers or email addresses will live forever in the databases of recruiters everywhere. PROTIP: Those that complain about recruiters often cite laziness in the initial contact. This may be evidenced by an obvious cut and paste or clear signs that the recruiter didn’t read the bio. If you want to screen out recruiters that don’t do the work, put up a barrier to weed out the lazy. This page that uses scripts in Python and Haskell to hide an email address is perhaps my favorite, but there are other less clever ways if you want to set the bar lower than the ability to cut/paste code into a compiler. Search Recruiters search for you, and you can search for them. Most recruiters are going to be easiest to find on LinkedIn due to the amount of time they spend there.Click on Advanced at the top of the main LinkedIn screen (just to the right of the search bar)  On the upper left side of your screen you will see several fields.  Make sure you are doing a People search (and not a Jobs search).Type ‘Recruiter’ and other terms specific to you in the Keywords field.  Try ‘developer‘ or ‘programmer’ and a term that a recruiter might use to brand you, such as a language.  Recruiters often populate their LinkedIn profiles with the technologies they seek, not unlike job seekers trying to catch the automated eye of a résumé scanning system.Enter the zip code of the area where you wish to find work and consider setting a mile limit.  Some recruiters work nationally, but local knowledge goes a long way if you are seeking to work in one area. Once you start entering the code, a menu appears.  Depending on where you live, you may want to select 25 or 50 miles (probably good for northeast or mid-Atlantic US), or up to 100 miles (for midwest).On the right, make sure you have 3rd + Everyone Else checked under Relationship.   This will maximize your results, particularly if your LinkedIn network is small.Click Search. Repeat, and vary the words you used in Step 3.  You should see a few different faces as you adjust the keywords, and you’ll also see whether you have connections in common with those in your search results.Twitter is another decent option. Make sure you are searching People (and not Everything), and pair up the word recruiter with some keywords and/or geographic locations. You’ll get numerous hits in most cases, and should only have to do a bit of legwork to find their bios. In addition to being able to find recruiters on social sites, you can use job boards as well. If you search for a Ruby job in New York City, you may quickly find that several of the listings are posted by one or two recruiting companies. Look into those firms to see if they have a specialty practice. Search engines might be a bit less useful and are likely to turn up the same listings found on job boards. Evaluating a Recruiter Once you have found a pool of potential recruiters, you need to decide which ones to contact. Most job seekers want a recruiter that can provide quality opportunities, has deep market knowledge, can leverage industry relationships, and will navigate issues in the hiring process. What criteria should we use in the evaluation? Experience Just like most disciplines, in recruiting there is no substitute for experience. It takes time to develop contacts and to learn how to uncover potential land mines. Extensive education, recruitment certifications, and training programs don’t get you a network or prepare you for handling unique situations. Early in my career I know I made many of the mistakes that technologists complain about, and I didn’t have a solid network or steady clients for at least five years. At a certain point in your recruiting career you may not have seen it all, but it’s rare that you are surprised by an outcome. Focus and Expertise Experienced recruiters that have spent little time in the industry may be good for general job search advice or negotiation, but can’t provide full value. Look for a consistent track record of years in your field and geography of your search. Talking to a few generalists will make the specialists stand out. Relationships/Clients Since recruiters aren’t paid by you and differ from a placement agency, it’s important that the firm has client relationships. Most firms do not advertise their client names which can make it difficult to discover the strength of an agency’s opportunities. The descriptions themselves could be enough to convince you that the agency has attractive clients. Agencies with solid relationships may reach out to past clients and contacts when they don’t have a position that is a clear fit for your background. Personality fit Being that an agency recruiter is going to be representing you to companies and even advocating and negotiating on your behalf, it’s important that you get along. You don’t need to be best friends, but someone who dislikes you is unlikely to fight for your best interests. A ten minute call should give the insight you need to make the decision. Ask questions about their experience and pay attention to the types of questions they ask you. If they don’t dig into your goals and objectives, they probably aren’t concerned with finding a good fit for you.Reference: So You Want to Use A Recruiter Part I – Recruit Your Recruiter from our JCG partner Dave Fecak at the Job Tips For Geeks blog....

Javascript for Java Developers

This post will go over the Javascript language from the point of view of a Java developer, focusing on the differences between the two languages and the frequent pain points. We will go over the following:                Objects Only, No Classes Functions are just Values The ‘this’ Keyword Classic vs Prototypal Inheritance Constructors vs Constructor Functions Closures vs Lambdas Encapsulation and Modules Block Scope and HoistingWhy Javascript in the Java World ? A lot of Java frontend development work is done using Java/XML based frameworks like JSF or GWT. The framework developers themselves need to know Javascript, but in principle the application developers don’t. However the reality is that:For doing custom component development in for example Primefaces (JSF), it’s important to know Javascript and jQuery. In GWT, integrating at least some third-party Javascript widgets is common and cost effective.The end result is that Javascript is usually needed to do at least the last 5 to 10% of frontend work, even using Java frameworks. Also it’s starting to get more and more used for polyglot enterprise development, alongside Angular for example. The good news is that, besides a few gotchas that we will get into, Javascript is a very learneable language for a Java developer. Objects Only – No Classes One of the most surprising things about Javascript is that although it’s an object oriented language, there are no classes (although the new Ecmascript 6 version will have them). Take for example this program, that initializes an empty object and set’s two properties: // create an empty object - no class was needed !! var superhero = {};superhero.name = 'Superman'; superhero.strength = 100; Javascript objects are just like a Java HashMap of related properties, where the keys are Strings only. The following would be the ‘equivalent’ Java code: Map<String,Object> superhero = new HashMap<>();superhero.put("name","Superman"); superhero.put("strength", 100); This means that a Javascript object is just a multi-level ‘hash map’ of key/value pairs, with no class definition needed. Functions Are Just Values Functions in Javascript are just values of type Function, it’s a simple as that! Take for example: var flyFunction = function() { console.log('Flying like a bird!'); };superhero.fly = flyFunction; This creates a function (a value of type Function) and assigns it to a variable flyFunction. A new property named fly is then created in the superhero object, that can be invoked like this: // prints 'Flying like a bird!' to the console superhero.fly(); Java does not have the equivalent of the Javascript Function type, but almost. Take for example the SuperHero class that takes a Power function: public interface Power { void use(); }public class SuperHero {private Power flyPower;public void setFly(Power flyPower) { this.flyPower = flyPower; }public void fly() { flyPower.use(); } } This is how to pass SuperHero a function in Java 7 and 8: // Java 7 equivalent Power flyFunction = new Power() { @Override public void use() { System.out.println("Flying like a bird ..."); } };// Java 8 equivalent superman.setFly( ()->System.out.println("Flying like a bird ..."));superman.fly(); So although a Function type does not exist in Java 8, this ends up not preventing a ‘Javascript-like’ functional programming style. But if we pass functions around, what happens to the meaning of the this keyword? The ‘this’ Keyword Usage What Javascript allows to do with this is quite surprising compared to the Java world. Let’s start with an example: var superman = {heroName: 'Superman',sayHello: function() { console.log("Hello, I'm " + this.heroName ); } };superman.sayHello(); This program creates an object superman with two properties: a String heroName and a Function named sayHello. Running this program outputs as expected Hello, I'm Superman. What if we pass the function around? By passing around sayHello, we can easily end up in a context where there is no heroName property: var failThis = superman.sayHello;failThis(); Running this snippet would give as output: Hello, I'm undefined. Why does this not work anymore? This is because the variable hello belongs to the global scope, which contains no member variable named heroName. To solve this:In Javascript the value of the this keyword is completely overridable to be anything that we want!// overrides 'this' with superman hello.call(superman); The snippet above would print again Hello, I'm Superman. This means that the value of this depends on both the context on which the function is called, and on how the function is called. Classic vs Prototypal Inheritance In Javascript, there is no class inheritance, instead objects can inherit directly from other objects. The way this works is that each object has an implicit property that points to a ‘parent’ object. That property is called __proto__, and the parent object is called the object’s prototype, hence the name Prototypal Inheritance. How does prototype work? When looking up a property, Javascript will try to find the property in the object itself. If it does not find it then it tries in it’s prototype, and so on. For example: var avengersHero = { editor: 'Marvel' };var ironMan = {};ironMan.__proto__ = avengersHero;console.log('Iron Man is copyrighted by ' + ironMan.editor); This snippet will output Iron Man is copyrighted by Marvel. As we can see, although the ironMan object is empty, it’s prototype does contain the property editor, which get’s found. How does this compare with Java inheritance? Let’s now say that the rights for the Avengers where bought by DC Comics: avengersHero.editor = 'DC Comics'; If we call ironMan.editor again, we now get Iron Man is copyrighted by DC Comics. All the existing object instances with the avengersHero prototype now see DC Comics without having to be recreated. This mechanism is very simple and very powerful. Anything that can be done with class inheritance can be done with prototypal inheritance. But what about constructors? Constructors vs Constructor Functions In Javascript an attempt was made to make object creation similar to languages like Java. Let’s take for example: function SuperHero(name, strength) { this.name = name; this.strength = strength; } Notice the capitalized name, indicating that it’s a constructor function. Let’s see how it can be used: var superman = new SuperHero('Superman', 100);console.log('Hello, my name is ' + superman.name); This code snippet outputs Hello, my name is Superman. You might think that this looks just like Java, and that is exactly the point! What this new syntax really does is to it creates a new empty object, and then calls the constructor function by forcing this to be the newly created object. Why is this syntax not recommended then? Let’s say that we want to specify that all super heroes have a sayHello method. This could be done by putting the sayHello function in a common prototype object: function SuperHero(name, strength) { this.name = name; this.strength = strength; }SuperHero.prototype.sayHello = function() { console.log('Hello, my name is ' + this.name); }var superman = new SuperHero('Superman', 100); superman.sayHello(); This would output Hello, my name is Superman. But the syntax SuperHero.prototype.sayHello looks anything but Java like! The new operator mechanism sort of half looks like Java but at the same time is completely different. Is there a recommended alternative to new? The recommended way to go is to ignore the Javascript new operator altogether and use Object.create: var superHeroPrototype = { sayHello: function() { console.log('Hello, my name is ' + this.name); } };var superman = Object.create(superHeroPrototype); superman.name = 'Superman'; Unlike the new operator, one thing that Javascript absolutely got right where Closures. Closures vs Lambdas Javascript Closures are not that different from Java anonymous inner classes used in a certain way. take for example the FlyingHero class: public interface FlyCommand { public void fly(); }public class FlyingHero {private String name;public FlyingHero(String name) { this.name = name; }public void fly(FlyCommand flyCommand) { flyCommand.fly(); } } We can can pass it a fly command like this in Java 8: String destination = "Mars"; superMan.fly(() -> System.out.println("Flying to " + destination )); The output of this snippet is Flying to Mars. Notice that the FlyCommand lambda had to ‘remember’ the variable destination, because it needs it for executing the fly method later. This notion of a function that remembers about variables outside it’s block scope for later use is called a Closure in Javascript. For further details, have a look at this blog post Really Understanding Javascript Closures. What is the main difference between Lambdas and Closures? In Javascript a closure looks like this: var destination = 'Mars';var fly = function() { console.log('Fly to ' + destination); }fly(); The Javascript closure, unlike the Java Lambda does not have the constraint that the destination variable must be immutable (or effectively immutable since Java 8). This seemingly innocuous difference is actually a ‘killer’ feature of Javascript closures, because it allows them to be used for creating encapsulated modules. Modules and Encapsulation There are no classes in Javascript and no public/ private modifiers, but then again take a look at this: function createHero(heroName) {var name = heroName;return { fly: function(destination) { console.log(name + ' flying to ' + destination); } }; } Here a function createHero is being defined, which returns an object which has a function fly. The fly function ‘remembers’ name when needed. How do Closures relate to Encapsulation? When the createHero function returns, noone else will ever be able to directly access name, except via fly. Let’s try this out: var superman = createHero('SuperMan');superman.fly('The Moon'); The output of this snippet is SuperMan flying to The Moon. But happens if we try to access name directly ? console.log('Hero name = ' + superman.name); The result is Hero name = undefined. The function createHero is said to a be a Javascript encapsulated module, with closed ‘private’ member variables and a ‘public’ interface returned as an object with functions. Block Scope and Hoisting Understanding block scope in Javascript is simple: there is no block scope! Take a look at this example: function counterLoop() {console.log('counter before declaration = ' + i);for (var i = 0; i < 3 ; i++) { console.log('counter = ' + i); }console.log('counter after loop = ' + i); }counterLoop(); By looking at this coming from Java, you might expect:error at line 3: ‘variable i does not exist’ values 0, 1, 2 are printed error at line 9: ‘variable i does not exist’It turns out that only one of these three things is true, and the output is actually this: counter before declaration = undefined counter = 0 counter = 1 counter = 2 counter after loop = 3 Because there is no block scope, the loop variable i is visible for the whole function. This means:line 3 sees the variable declared but not initialized line 9 sees i after the loop has terminatedWhat might be the most puzzling is that line 3 actually sees the variable declared but undefined, instead of throwing i is not defined. This is because the Javascript interpreter first scans the function for a list of variables, and then goes back to interpret the function code lines one by one. The end result is that it’s like the variable i was hoisted to the top, and this is what the Javascript runtime actually ‘sees’: function counterLoop() {var i; // i is 'seen' as if declared here!console.log('counter before declaration = ' + i);for (i = 0; i < 3 ; i++) { console.log('counter = ' + i); }console.log('counter after loop: ' + i); } To prevent surprises caused by hoisting and lack of block scoping, it’s a recommended practice to declare variables always at the top of functions. This makes hoisting explicit and visible by the developer, and helps to avoid bugs. The next version of Javascript (Ecmascript 6) will include a new keyword ‘let’ to allow block scoping. Conclusion The Javascript language shares a lot of similarities with Java, but also some huge differences. Some of the differences like inheritance and constructor functions are important, but much less than one would expect for day to day programming. Some of these features are needed mostly by library developers, and not necessarily for day to day application programming. This is unlike some of their Java counterparts which are needed every day. So if you are hesitant to give it a try, don’t let some of these features prevent you from going further into the language. One thing is for sure, at least some Javascript is more or less inevitable when doing Java frontend development, so it’s really worth to give it a try.Reference: Javascript for Java Developers from our JCG partner Aleksey Novik at the The JHades Blog blog....

MineCraft and off heap memory

Overview MineCraft is a really good example of when off heap memory can really help.The key requirements are:The bulk of the retained data is a simple data structure (in minecraft’s case its lots of byte[]) The usage of off heap memory can be hidden in abstraction.    The testI used the following test for starting minecraft server from a seed from scratch which is a particularly expensive operation for the server.Preset the level-seed=114 in server.properties Delete the world* directories Start the server with these options to see what the GC is doing -Xloggc:gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -Xmx6g Connect with one client Perform /worldgen village Perform /save-all Exit.To analyse the logs I am using jClarity’s Censum.Standard MineCraft There are two particularly expensive things it does:It caches block stage in many byte[]s It attempts to cache int[] used for processing without limit.A Censum report for the above test looks like this:The high pause times are partly due to having to manage the large objects.Off heap MineCraft Two changes were made to address this:Use off heap ByteBuffer for long term caching.  Unsafe would be more efficient but not as portable. Put a cap on the number of int[] cached.Note: the problem with the temporary int[] only became visible to me after moving the bulk of the data off heap. Addressing the biggest problem over reveals more quick fix problems.A Censum report for the same test looks like this: There is still some premature promotion i.e. further improvements can be made but you can see that the application is spending.ConclusionUsing off heap memory can help you tame you GC pause times, especially if the bulk of your data is in simple data structures which can be easily abstracted.  Doing so can also help reveal other simple optimisations you can do to improve the consistency of your performance. FootnoteMany organisations treat performance as an enhancement and optional, however if you instil a culture where reasonable performance is a requirement, and failing to meet this requirement is a bug, performance issues are more likely to be fixed. c.f. https://bugs.mojang.com/browse/MC-56447 The Source UsedThe source used for the test is available here. https://github.com/peter-lawrey/MineOffHeapThe logs produced are available here. https://github.com/peter-lawrey/MineOffHeap/tree/master/logsReference: MineCraft and off heap memory from our JCG partner Peter Lawrey at the Vanilla Java blog....

Java Build Tools: Ant vs Maven vs Gradle

In the beginning there was Make as the only build tool available. Later on it was improved with GNU Make. However, since then our needs increased and, as a result, build tools evolved. JVM ecosystem is dominated with three build tools:Apache Ant with Ivy Maven Gradle  Ant with Ivy Ant was the first among “modern” build tools. In many aspects it is similar to Make. It was released in 2000 and in a short period of time became the most popular build tool for Java projects. It has very low learning curve thus allowing anyone to start using it without any special preparation. It is based on procedural programming idea. After its initial release, it was improved with the ability to accept plug-ins. Major drawback was XML as the format to write build scripts. XML, being hierarchical in nature, is not a good fit for procedural programming approach Ant uses. Another problem with Ant is that its XML tends to become unmanageably big when used with all but very small projects. Later on, as dependency management over the network became a must, Ant adopted Apache Ivy. Main benefit of Ant is its control of the build process. Maven Maven was released in 2004. Its goal was to improve upon some of the problems developers were facing when using Ant. Maven continues using XML as the format to write build specification. However, structure is diametrically different. While Ant requires developers to write all the commands that lead to the successful execution of some task, Maven relies on conventions and provides the available targets (goals) that can be invoked. As the additional, and probably most important addition, Maven introduced the ability to download dependencies over the network (later on adopted by Ant through Ivy). That in itself revolutionized the way we deliver software. However, Maven has its own problems. Dependencies management does not handle well conflicts between different versions of the same library (something Ivy is much better at). XML as the build configuration format is strictly structured and highly standardized. Customization of targets (goals) is hard. Since Maven is focused mostly on dependency management, complex, customized build scripts are actually harder to write in Maven than in Ant. Maven configuration written in XML continuous being big and cumbersome. On bigger projects it can have hundreds of lines of code without actually doing anything “extraordinary”. Main benefit from Maven is its life-cycle. As long as the project is based on certain standards, with Maven one can pass through the whole life cycle with relative ease. This comes at a cost of flexibility. In the mean time the interest for DSLs (Domain Specific Languages) continued increasing. The idea is to have languages designed to solve problems belonging to a specific domain. In case of builds, one of the results of applying DSL is Gradle. Gradle Gradle combines good parts of both tools and builds on top of them with DSL and other improvements. It has Ant’s power and flexibility with Maven’s life-cycle and ease of use. The end result is a tool that was released in 2012 and gained a lot of attention in a short period of time. For example, Google adopted Gradle as the default build tool for the Android OS. Gradle does not use XML. Instead, it had its own DSL based on Groovy (one of JVM languages). As a result, Gradle build scripts tend to be much shorter and clearer than those written for Ant or Maven. The amount of boilerplate code is much smaller with Gradle since its DSL is designed to solve a specific problem: move software through its life cycle, from compilation through static analysis and testing until packaging and deployment. It is using Apache Ivy for JAR dependencies. Gradle effort can be summed as “convention is good and so is flexibility”. Code examples We’ll create build scripts that will compile, perform static analysis, run unit tests and, finally, create JAR files. We’ll do those operations in all three frameworks (Ant, Maven and Gradle) and compare the syntax. By comparing the code for each task we’ll be able to get a better understanding of the differences and make an informed decision regarding the choice of the build tool. First things first. If you’ll do the examples from this article by yourself, you’ll need Ant, Ivy, Maven and Gradle installed. Please follow installation instructions provided by makers of those tools. You can choose not to run examples by yourself and skip the installation altogether. Code snippets should be enough to give you the basic idea of how each of the tools work. Code repository https://github.com/vfarcic/JavaBuildTools contains the java code (two simple classes with corresponding tests), checkstyle configuration and Ant, Ivy, Maven and Gradle configuration files. Let’s start with Ant and Ivy. Ant with Ivy Ivy dependencies need to be specified in the ivy.xml file. Our example is fairly simple and requires only JUnit and Hamcrest dependencies. [ivy.xml] <ivy-module version="2.0"> <info organisation="org.apache" module="java-build-tools"/> <dependencies> <dependency org="junit" name="junit" rev="4.11"/> <dependency org="org.hamcrest" name="hamcrest-all" rev="1.3"/> </dependencies> </ivy-module> Now we’ll create our Ant build script. Its task will be only to compile a JAR file. The end result is the following build.xml. [build.xml] <project xmlns:ivy="antlib:org.apache.ivy.ant" name="java-build-tools" default="jar"><property name="src.dir" value="src"/> <property name="build.dir" value="build"/> <property name="classes.dir" value="${build.dir}/classes"/> <property name="jar.dir" value="${build.dir}/jar"/> <property name="lib.dir" value="lib" /> <path id="lib.path.id"> <fileset dir="${lib.dir}" /> </path><target name="resolve"> <ivy:retrieve /> </target><target name="clean"> <delete dir="${build.dir}"/> </target><target name="compile" depends="resolve"> <mkdir dir="${classes.dir}"/> <javac srcdir="${src.dir}" destdir="${classes.dir}" classpathref="lib.path.id"/> </target><target name="jar" depends="compile"> <mkdir dir="${jar.dir}"/> <jar destfile="${jar.dir}/${ant.project.name}.jar" basedir="${classes.dir}"/> </target></project> First we specify several properties. From there on it is one task after another. We use Ivy to resolve dependencies, clean, compile and, finally, create the JAR file. That is quite a lot of configuration for a task that almost every Java project needs to perform. To run the Ant task that creates the JAR file, execute following. ant jar Let’s see how would Maven does the same set of tasks. Maven [pom.xml] <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0http://maven.apache.org/maven-v4_0_0.xsd"><modelVersion>4.0.0</modelVersion> <groupId>com.technologyconversations</groupId> <artifactId>java-build-tools</artifactId> <packaging>jar</packaging> <version>1.0</version><dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.11</version> </dependency> <dependency> <groupId>org.hamcrest</groupId> <artifactId>hamcrest-all</artifactId> <version>1.3</version> </dependency> </dependencies><build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>2.3.2</version> </plugin> </plugins> </build></project> To run the Maven goal that creates the JAR file, execute following. mvn package The major difference is that with Maven we don’t need to specify what should be done. We’re not creating tasks but setting the parameters (what are the dependencies, what plugins to use…). This shows the major difference between Ant and Maven. Later promotes the usage of conventions and provides goals (targets) out-of-the-box. Both Ant and Maven XML files tend to grow big with time. To illustrate that, we’ll add Maven CheckStyle, FindBugs and PMD plugins that will take care of static analysis. All three are fairly standard tools used, in one form or another, in many Java projects. We want all static analysis to be executed as part of a single target verify together with unit tests. Moreover, we should specify the path to the custom checkstyle configuration and make sure that it fails on error. Additional Maven code is following: [pom.xml] <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-checkstyle-plugin</artifactId> <version>2.12.1</version> <executions> <execution> <configuration> <configLocation>config/checkstyle/checkstyle.xml</configLocation> <consoleOutput>true</consoleOutput> <failsOnError>true</failsOnError> </configuration> <goals> <goal>check</goal> </goals> </execution> </executions> </plugin> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>findbugs-maven-plugin</artifactId> <version>2.5.4</version> <executions> <execution> <goals> <goal>check</goal> </goals> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-pmd-plugin</artifactId> <version>3.1</version> <executions> <execution> <goals> <goal>check</goal> </goals> </execution> </executions> </plugin> To run the Maven goal that runs both unit tests and static analysis with CheckStyle, FindBugs and PMD, execute following. mvn verify We had to write a lot of XML that does some very basic and commonly used set of tasks. On real projects with a lot more dependencies and tasks, Maven pom.xml files can easily reach hundreds or even thousands of lines of XML. Here’s how the same looks in Gradle. Gradle [build.gradle] apply plugin: 'java' apply plugin: 'checkstyle' apply plugin: 'findbugs' apply plugin: 'pmd'version = '1.0'repositories { mavenCentral() }dependencies { testCompile group: 'junit', name: 'junit', version: '4.11' testCompile group: 'org.hamcrest', name: 'hamcrest-all', version: '1.3' } Not only that the Gradle code is much shorter and, to those familiar with Gradle, easier to understand than Maven, but it actually introduces many useful tasks not covered with the Maven code we just wrote. To get the list of all tasks that Gradle can run with the current configuration, please execute the following. gradle tasks --all Clarity, complexity and the learning curve For newcomers, Ant is the clearest tool of all. Just by reading the configuration XML one can understand what it does. However, writing Ant tasks easily gets very complex. Maven and, specially, Gradle have a lot of tasks already available out-of-the-box or through plugins. For example, by seeing the following line it is probably not clear to those not initiated into mysteries of Gradle what tasks will be unlocked for us to use. [build.gradle] apply plugin: 'java' This simple line of code adds 20+ tasks waiting for us to use. Ant’s readability and Maven’s simplicity are, in my opinion, false arguments that apply only during the short initial Gradle learning curve. Once one is used to the Gradle DSL, its syntax is shorter and easier to understand than those employed by Ant or Maven. Moreover, only Gradle offers both conventions and creation of commands. While Maven can be extended with Ant tasks, it is tedious and not very productive. Gradle with Groovy brings it to the next level. Next article will go deeper into Gradle and explain in more details its integration with Groovy.Reference: Java Build Tools: Ant vs Maven vs Gradle from our JCG partner Viktor Farcic at the Technology conversations blog....

NoSQL on Android

There are various NoSQL solutions for mobile platforms such as the iOS and Android. Here, we will look at Couchbase Lite (CBL – the successor of TouchDB), a lightweight, full-featured, embedded JSON database. Why a full-feature database instead of just a wrapper for a cloud service? Basically, responsiveness. The idea is that the application should always be available to users even when the network is down or slow. Of course, being able to work with data locally also means that we’ll have to sync with the server at some point. Note that CBL is significantly different from Apache CouchDB despite the “Couch” part. Now, let’s get down to business and create a small Android application (entire code available on GitHub) which sets up a database using CBL’s Native API, and run basic CRUD operations. After going through a relatively easy setup (this one is for good old Eclipse users, that one for Android Studio fans), let’s start by writing some code: import com.couchbase.lite.*; import com.couchbase.lite.android.AndroidContext;/** Database wrapper*/ public class CbDatabase {private Database database; private Manager manager;/** Ctor Setup */ public CbDatabase(String dbname) throws IOException, CouchbaseLiteException { // 1. use default settings (read/write access) manager = new Manager( new AndroidContext(ctx), Manager.DEFAULT_OPTIONS );// 2. Check database name given by user // No upper case allowed in CBL! // Only the following characters are valid: // abcdefghijklmnopqrstuvwxyz0123456789_$()+-/if ( ! Manager.isValidDatabaseName(dbname)) { // report... return; } // 3. Get existing db with that name // or create a new one if it doesn't exist database = manager.getDatabase(dbname); }//...more methods to come } The code above uses a Manager to set up a database with a valid name. A CBL Database is basically a container for documents. A Manager can create several different databases, each with its own namespace. Here’s how to release all the resources: /** */ public void close(){ if(manager != null){ manager.close(); } } Once given the Database object, we are now ready to do CRUD operations on it: /** C-rud */ public String create( Map<String, Object> docContent ) throws CouchbaseLiteException {// create an empty document Document doc = database.createDocument(); // add content to document and save it doc.putProperties(docContent); return doc.getId(); } A CBL Document is the primary entity stored in a database, and has the following attributes:A unique ID (_id property) which can be automatically generated as a UUID A revision ID (_rev property) to keep track of updates A set of key/value pairs forming the body of the DocumentOnce created, we can retrieve a Document’s content by its id: /** c-R-ud */ public Map<String, Object> retrieve(String docId){ return database.getDocument(docId) .getProperties(); } We can update an existing key/value pair entry in a Document by using a callback: /** cr-U-d (auto-retry) */ public void update( final String key, final Object value, String docId ) throws CouchbaseLiteException {// retrieve the document from the database Document doc = database.getDocument(docId); doc.update(new Document.DocumentUpdater() { @Override /** This may be called more than once */ public boolean update(UnsavedRevision newRevision) { Map<String, Object> properties = newRevision.getUserProperties(); properties.put(key, value); newRevision.setUserProperties(properties); return true; } }); } The method retries automatically the update in case of write conflicts by re-reading the Document and re-calling the callback method. That process will continue until the write succeeds. As its name implies, UnsavedRevision is a mutable copy to work with until data is finally persisted to the database. Deleting a Document is pretty straightforward. Here, we do it by its id: /** cru-D */ public boolean delete(String docId) throws CouchbaseLiteException {// retrieve the document from the database Document doc = database.getDocument(docId); // delete the document doc.delete(); return doc.isDeleted(); } Deleting the Document will create a new revision called the “tombstone” (no kidding). It will be marked as deleted so that its status can be replicated to a server when doing a sync. As for updates, this can lead to conflicts if other users made changes to the same Document. In that case, the decision to retry the delete is left to us. We are now ready to use this wrapper class in our project. First, let’s define our use case: let’s say we need to store locally some user data for our mobile games, like the user email, when the user registered in our system, and a bunch of game scores. Here’s a quick test we can run in our main Activity’s onCreate() method. First, let’s input some data: // get the current date and time Date now = new Date(); String nowString = DateFormat.getDateTimeInstance( DateFormat.LONG, DateFormat.LONG).format(now); // game scores List<Double> scores = new ArrayList<Double>(); scores.add(190.00); scores.add(210.00); scores.add(250.00); scores.add(275.00); // create an object that contains data for a document Map<String, Object> docContent = new HashMap<String, Object>(); docContent.put("email", "Nomad@nomad.com"); docContent.put("registered", nowString); docContent.put("scores", scores); Now, the database operations: try{ CbDatabase db = new CbDatabase("testdb"); // 1. Create String docId = db.create(docContent); // 2. Retrieve Map<String, Object> docContent = db.retrieve(docId); // 3. Update scores.add(350.00); db.update("scores", scores, docId); // 4. Delete boolean deleted = db.delete(docId); assert(deleted == true); } catch (Exception e) { // handle here... } finally{ if(db != null){ db.close(); } } Here are the corresponding JSON Documents for Retrieve and Update: // retrieved { _rev=1-1ef4c4618a712cdf437d4b0c92594ddc, _id=69fdcb83-1774-4a3f-9e88-b298d3c7304a, scores=[190.0, 210.0, 250.0, 275.0], email=Nomad@nomad.com, registered=June 18, 2014 11:03:18 AM GMT+02:00 } //-------------------------------------------- // updated scores: note the change in revision { _rev=2-23e6d83d243f80c03b17c4160d511e16, scores=[190.0, 210.0, 250.0, 275.0, 350.0], _id=69fdcb83-1774-4a3f-9e88-b298d3c7304a, email=Nomad@nomad.com, registered=June 18, 2014 11:03:18 AM GMT+02:00 } That’s it. Further code details are on GitHub. We just scratched the surface of CBL with this very basic introduction. Unfortunately, the CBL online documentation for Android is incomplete at this time, compared to the one for the iOS. Anyway, there are other important CBL features like adding attachments, Views & Queries, replication (Server & P2P), a REST API, etc. Those would be the subjects of future articles.Reference: NoSQL on Android from our JCG partner Tony Sicilian at the Tony’s Blog blog....

Java 8 Lambdas vs Groovy Closures Compactness: Grouping And Summing

Java 8 is featuring lambdas, which are similar to a construction Groovy has already for some time: closures. In Groovy we could already do this:               def list = ['a', 'b', 'c'] print list.collect { it.toUpperCase() } // [A, B, C] where { it.toUpperCase() } is the closure. In Java 8 we can achieve the same functionality now in a concise way. list.stream().map( s -> s.toUpperCase() ) Although you could argue that with proper use of the new Stream API, bulk operations and method references, at least the intent of a piece of code is conveyed more clearly now – Java’s verboseness can still cause sore eyes.Here are some other examples. Some Groovy animals class Animal { String name BigDecimal price String farmer String toString() { name } }def animals = [] animals << new Animal(name: "Buttercup", price: 2, farmer: "john") animals << new Animal(name: "Carmella", price: 5, farmer: "dick") animals << new Animal(name: "Cinnamon", price: 2, farmer: "dick")   Example 1: Summing the total price of all animals Groovy assert 9 == animals.sum { it.price } // or animals.price.sum() What Groovy you see here:sum can be called on a List and optionally passed a closure defining the property of “it” – the animal being iterated over – to sort on. or sum can be called on a List without any arguments, which is equivalent to invoking the “plus” method on all items in the collection.Java 8 Optional<BigDecimal> sum = animals. stream(). map(Animal::getPrice). reduce((l, r) -> l.add(r)); assert BigDecimal.valueOf(9) == sum.get(); What Java you see here:Through the Stream API’s stream method we can create a pipeline of operations, such as map and reduce The argument to the map operation is a method reference to the getPrice() method of the currently iterated animal. We could also replace this part with the expression a -> a.getPrice() reduce is a general reduction operation (also called a fold) in which the BigDecimals of the prices are added up. This is also giving us an Optional with the total sum. BTW, if we were to use a double for price – which we don’t because I want to give a good example – we could have used an existing DoubleStream with a sum() on it e.g. double sum = animals.stream().mapToDouble(Animal::getPrice).sum();  Example 2: Grouping all animals by farmer Groovy def animalsByFarmer = animals.groupBy { it.farmer } // [john:[Buttercup], dick:[Carmella, Cinnamon]] Java 8 Map<String, List<Animal>> animalsByFarmer = animals .stream() .collect( Collectors.groupingBy(Animal::getFarmer)); // {dick=[Carmella, Cinnamon], john=[Buttercup]}   Example 3: Summing the total price of all animals grouped by farmer Groovy def totalPriceByFarmer = animals .groupBy { it.farmer } .collectEntries { k, v -> [k, v.price.sum()] } // [john:2, dick:7] What Groovy you see here:collectEntries iterates through the “groupBy” map transforming each map entry using the k, v -> ... closure returning a map of the transformed entries. v.price is actually a List of prices (per farmer) – such as in example 1 – on which we can call sum().Java 8 Map<String, BigDecimal> totalPriceByFarmer = animals .stream() .collect( Collectors.groupingBy( Animal::getFarmer, Collectors.reducing( BigDecimal.ZERO, Animal::getPrice, BigDecimal::add))); // {dick=7, john=2} This Java code again yields the same results. Since IDE’s, Eclipse at least, don’t format this properly, you’ll have to indent these kinds of constructions for readability a bit yourself.Reference: Java 8 Lambdas vs Groovy Closures Compactness: Grouping And Summing from our JCG partner Ted Vinke at the Ted Vinke’s Blog blog....

SpringSource Certified Spring Professional

When I first heard about this certification I was really excited and wanted to find out what I can about this exam before I start any initiative to pass it. What I found however was the complete lack of resources, guides, information or mock exams of any kind. The only reasonable resource online I was able to find before my exam was a blog post about certification experiences by Jeanne Boyarsky on her well-known blog. However these information are almost 3 years old and few things have changed since then, so please allow me to update you on the current state of the certification process. The first thing that struck me was the ordering process before any certification-related stuff even started. The whole ordering process was really long and required extensive email communication with VMware’s support (support people were actually quite helpful and made it more bearable). Given my employer at a time was in VMware’s partner program I don’t know how the experience is for regular customers. It might be caused by recent VMware’s acquisition of SpringSource. To paint a complete picture I was ordering the certification for me and two other colleagues. So beware, there might be dragons there. About certification Back in a day, there were two possible ways of getting certified: by attending the course or being a grandfathered candidate. Grandfathering is no longer allowed so the only way to get the certification is to attend the course and afterwards pass the exam. When you register for the course you pay about 1 375 € (around 1 900 $). This price covers the course (+ the course-ware) and one free exam attempt. Any following retake attempt costs 150 $. Unlike with Oracle’s Java Programmer certification there are no requirements for prior certification/course (besides experience with Java development). The course Mandatory part of the certification is a 4 day course that focuses on following topics:Introduction to Spring Advanced XML Dependency Injection Annotation-Based Dependency Injection Java-Based Dependency Injection Bean Life Cycle: How Does Spring Work Internally? Testing a Spring-Based Application Aspect-Oriented Programming Data Access and JDBC with Spring Database Transactions with Spring Integrating Spring with JPA and Hibernate Spring in a Web Application (MVC) Spring Security Advanced Topics (remoting, JMS, JMX)Each attendee gets a starter kit depicted below. It contains Core Spring Lecture Manual full of slides presented throughout the course. Most of the chapters are required for the certification however the last few chapters are there to help you understand some of the related issues to main topics. The second book you will receive is a notebook for your notes and pictures/diagrams. These two books come with USB stick that contains all materials in digital form as well as all labs that are necessary to complete the course. Tablet is not part of the kit, but I can highly recommend using it if you have one, since all the slides are available in PDF form.  Please note that the course is prepared the way that even person with no prior Spring experience can complete it with proper studying before the exam. Having only a year worth of experience with Spring when I started the order process I was really glad that even if you have never worked with for example AOP you could practice the exercises prepared in labs. The course is divided 50% lectures and 50% lab time. This brings me to labs – a series of workspaces. Each workspace comes in two favors: incomplete/non-functional workspace and fully working workspace providing a solution to all problems simulated in the previous one. This is an essential part of the course since you get a hands-on experience with all of the topics. Course is led by an experienced Spring lecturer that is able to answer any question and if not they are willing to try out / simulate any twisted idea you are able to think of (based on the course I attended). I must say that the course was a really pleasant experience for me. I found the materials easy to work with (mostly based on the fact that their business domain remains the same throughout the course). Even if you happen to be an experienced Spring user there are always some aspects of Spring framework you might (willingly) have no experience with. The course provides you with enough theory and examples to gain basic orientation in given area. The ultimate goal of your attendance should be a state when you understand everything in your course book. If you feel uncertain, ask questions – course trainer will surely explain any problematic areas. The exam Exam is 90 minutes long including 2 minutes for legal agreements and stuff like this. You are expected to answer 50 questions in 88 minutes which I found plenty of time (it took me about an hour with one final revision at the end). Passing score for the exam is 76%. Always read every single word in a question very carefully. Some questions ask which of the following is/is not correct and you have to be absolutely sure that you know what you are asked to do. There were only single choice and multiple choice questions. Most questions were addressing container and testing, AOP and transactions (smaller number of questions from other areas). There are no questions about REST (contrary to the Jeanne’s version of test). Don’t expect tricky and syntax heavy questions like in Oracle’s Java Programmer certification. Questions are straightforward and you either know the answer or not. Exam software allows you to browse questions, mark them for a review and final review. You will receive the results of your exam right after submitting your answers. You will learn your overall score, section scores and final grade. Preparation When it comes to the exam preparation I found 3 weeks to be enough to get myself prepared for the exam (passing with 94%). My first move was to ditch course-ware for a week and hit a well-written book about Spring called Spring in Action, Third Edition by Craig Walls. Having read first and second parts of the book (excluding chapter about Spring Web Flow) I gained a solid understanding of all mentioned areas of Spring. This was my first week of studies after the course. After this I decided to study official Spring documentation to gain even more insight into the container capabilities (not necessary for the exam, but definitely worth reading for real world applications – there is no such thing as over-studying). As final step I did all the labs and went through the course book. As an additional resources I would like to include works of others sharing their experience with this certification. My study plan:1 week4 day course2 weekRead first two parts of Spring in Action, Third Edition3 weekRead most of The IoC container documentation4 weekRead course book and did all the labs5 weekTook the examHelpful notes from fellow bloggers:jeanne’s core spring 3 certification experiences After the Spring Core 3.0 Exam Spring Professional Certification Exam from Java CodeBookStudy notes Even though the course-ware combined with Spring in Action, official documentation and Jeanne’s study notes provided quite a robust preparation material I felt the need to create a small addition to these. There were certain areas that needed to be cleared for me so I did my own addition to those study notes. It is rather brief and short (based on the fact I had quite a lot of material already). However I find them to be a valuable resource since they provide big picture of certain areas of Spring.My study notesCertificate I find it slightly disappointing that VMware is not providing a hard-copy of the certificate. So do not expect to receive any mail from them. The only thing you will receive is a soft-copy that you yourself can print out. Since I am still waiting for mine I can’t provide more information.(update 12.04.2014) It seems that unless you explicitly ask for PDF version of your certificate you won’t receive anything. So I did write to VMware’s certification support, provided all documents and scores regarding my exam and got my certificate in a day or two. Conclusion Definitely a positive experience – all the way. When I started with the certification process my Spring knowledge was quite limited and I was lacking understanding of some core Spring principles and concepts. However after the course I felt more confident and grew my knowledge by studying resources introduced in previous chapters. I can honestly say that now I am using many things learned there in my daily work. It is rather expensive certification, but definitely worth the trouble. So if you are planning on taking the exam I wish you good luck and hope this post helps you on your way to becoming SpringSource Certified Spring Professional.Reference: SpringSource Certified Spring Professional from our JCG partner Jakub Stas at the Jakub Stas blog....

An Alternative to the Twitter River – Index Tweets in Elasticsearch with Logstash

For some time now I’ve been using the Elasticsearch Twitter river for streaming conference tweets to Elasticsearch. The river runs on an Elasticsearch node, tracks the Twitter streaming API for keywords and directly indexes the documents in Elasticsearch. As the rivers are about to be deprecated it is time to move on to the recommended replacement: Logstash. With Logstash the retrieval of the Twitter data is executed in a different process, probably even on a different machine. This helps in scaling Logstash and Elasticsearch seperately.     Installation The installation of Logstash is nearly as easy as the one for Elasticsearch though you can’t start it without a configuration that tells it what you want it to do. You can download it, unpack the archive and there are scripts to start it. If you are fine with using the embedded Elasticsearch instance you don’t even need to install this separately. But you need to have a configuration file in place that tells Logstash what to do exactly. Configuration The configuration for Logstash normally consists of three sections: The input, optional filters and the output section. There is a multitude of existing components for each of those available. The structure of a config file looks like this (taken from the documentation): # This is a comment. You should use comments to describe # parts of your configuration. input { ... }filter { ... }output { ... } We are using the Twitter input, the elasticsearch_http output and no filters. Twitter As with any Twitter API interaction you need to have an account and configure the access tokens. input { twitter { # add your data consumer_key => "" consumer_secret => "" oauth_token => "" oauth_token_secret => "" keywords => ["elasticsearch"] full_tweet => true } } You need to pass in all the credentials as well as the keywords to track. By enabling the full_tweet option you can index a lot more data, by default there are only a few fields and interesting information like hashtags or mentions are missing. The Twitter river seems to have different names than the ones that are sent with the raw tweets so it doesn’t seem to be possible to easily index Twitter logstash data along with data created by the Twitter river. But it should be no big deal to change the Logstash field names as well with a filter. Elasticsearch There are three plugins that are providing an output to Elasticsearch: elasticsearch, elasticsearch_http and elasticsearch_river. elasticsearch provides the opportunity to bind to an Elasticsearch cluster as a node or via transport, elasticsearch_http uses the HTTP API and elasticsearch_river communicates via the RabbitMQ river. The http version lets you use different Elasticsearch versions for Logstash and Elasticsearch, this is the one I am using. Note that the elasticsearch plugin also provides an option for setting the protocol to http that also seems to work. output { elasticsearch_http { host => "localhost" index => "conf" index_type => "tweet" } } In contrast to the Twitter river the Logstash plugin does not create a special mapping for the tweets. I didn’t go through all the fields but for example the coordinates don’t seem to be mapped correctly to geo_point and some fields are analyzed that probably shouldn’t be (urls, usernames). If you are using those you might want to prepare your index by supplying it with a custom mapping. By default tweets will be pushed to Elasticsearch every second which should be enough for any analysis. You can even think about reducing this with the property idle_flush_time. Running Finally, when all of the configuration is in place you can execute Logstash using the following command (assuming the configuration is in a file twitter.conf): bin/logstash agent -f twitter.conf Nothing left to do but wait for the first tweets to arrive in your local instance at http://localhost:9200/conf/tweet/_search?q=*:*pretty=true. For the future it would be really useful to prepare a mapping for the fields and a filter that removes some of the unused data. For now you have to check what you would like to use of the data and prepare a mapping in advance.Reference: An Alternative to the Twitter River – Index Tweets in Elasticsearch with Logstash from our JCG partner Florian Hopf at the Dev Time blog....
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books