Featured FREE Whitepapers

What's New Here?

json-logo

Develop Restful web services using Spring MVC

REST INTRODUCTIONFrom Wikipedia: REST-style architectures consist of clients and servers. Clients initiate requests to servers; servers process requests and return appropriate responses. Requests and responses are built around the transfer of representations of resources. A resource can be essentially any coherent and meaningful concept that may be addressed.As you have read the most important thing in Rest architecture is the existance of a resource. This resource  can be anything (typically required information requested by client) that can be identified with a global identifier (URI in case of HTTP). In order to manipulate these resources, client communicates using standard interfaces (like HTTP) and exchange representations of these resources (using HTML, XML, …).Note that Rest does not force you to use any specific network protocol nor how resources are identified. For those who have never read about Rest this description of Rest architecture could seem something strange and bit complicated. A RESTful web service is a simple web service implemented using HTTP and the principles of REST. URI is defined as global identified, communication interface is HTTP and resource representation can be any valid Internet media type like JSON, XML or YAML. The set of operations that can be executed to resources depend on HTTP Methods and are (GET – retrieving/listing, PUT – replacing/updating, POST – creating and DELETE – deleting).HANDS ON WORKLet’s create our first Rest application with help of Spring MVC. Imagine an application that has a database of manga characters, and you want to provide a Rest interface so clients can retrieve characters following a RESTful strategy.First thing to do is identify the resource. In this case it is easy, “a character“. Next step is finding a URI that determines unequivocally a character. Easy too de facto rule can be applied here. This rule suggests that a unique URI can be <host>/<applicationname>/<resourceName>s/<id> in our case to return (GET) character with id 1 the URI would be “http://localhost:8080/RestServer/characters/1“. If no identifier is present all characters should be retrieved. If instead of GET, POST is used, a character with id “1″ would be inserted. And finally decide which Internet media type is required, in this case doesn’t matter because we are implementing both client and server so initially XML will be used.CODING Let’s start with a simple Spring MVC application created with Spring MVC template. Not much secret here, you will have a servlet-context.xml where component-scan, annotation-driven and InternalResourceViewResolver are registered. <?xml version="1.0" encoding="UTF-8" ?> <beans:beans xmlns="http://www.springframework.org/schema/mvc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:beans="http://www.springframework.org/schema/beans" xmlns:context="http://www.springframework.org/schema/context" xmlns:util="http://www.springframework.org/schema/util" xsi:schemaLocation="http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-3.0.xsd http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd"> <!-- DispatcherServlet Context: defines this servlet's request-processing infrastructure --> <!-- Enables the Spring MVC @Controller programming model --><annotation-driven /><!-- Resolves views selected for rendering by @Controllers to .jsp resources in the /WEB-INF/views directory --><beans:bean class="org.springframework.web.servlet.view.InternalResourceViewResolver"> <beans:property name="prefix" value="/WEB-INF/views/" /> <beans:property name="suffix" value=".jsp" /> </beans:bean><context:component-scan base-package="org.springframework.rest" /> </beans:beans>Next step is defining Character class. A simple POJO with four attributes. Class is converted to its XML representation using Jaxb annotation. Jaxb allows developers to map Java classes to XML representations and viceversa. package org.springframework.rest;import java.net.URL;import org.codehaus.jackson.annotate.JsonAutoDetect;@XmlRootElement public final class Character {private int id; private String name; private boolean isHuman; private URL characterUrl;protected Character() {}public Character(int id, String name, boolean isHuman, URL characterUrl) { super(); this.id = id; this.name = name; this.isHuman = isHuman; this.characterUrl = characterUrl; }public int getId() { return id; }public void setId(int id) { this.id = id; }public String getName() { return name; }public void setName(String name) { this.name = name; }public boolean isHuman() { return isHuman; }public void setHuman(boolean isHuman) { this.isHuman = isHuman; }public URL getCharacterUrl() { return characterUrl; }public void setCharacterUrl(URL characterUrl) { this.characterUrl = characterUrl; }@Override public int hashCode() { final int prime = 31; int result = 1; result = prime * result + id; return result; }@Override public boolean equals(Object obj) { if (this == obj) return true; if (obj == null) return false; if (getClass() != obj.getClass()) return false; Character other = (Character) obj; if (id != other.id) return false; return true; }}And finally the most important class in Spring MVC, “The Controller“. Controller will be the responsible of implementing required operations of Character resource. In current case only GET is implemented, the other operations would be similar. Let’s see the code: @Controller public class HomeController {private static final Map<Integer, Character> characters = new HashMap<Integer, Character>();static { try { characters.put(1, new Character(1, "Totoro", false, new URL("http://animeonly.org/albums/VISINAUJI/EGIO/fourth/Mon-Voisin-Totoro/normal_totoro_001.jpg"))); characters.put(2, new Character(2, "Satsuki Kusakabe", true, new URL("http://profile.ak.fbcdn.net/hprofile-ak-ash2/48980_1802552968_7286_n.jpg"))); characters.put(3, new Character(3, "Therru", false, new URL("http://28.media.tumblr.com/tumblr_lj4ctjKA8Y1qdvyqpo1_400.jpg"))); } catch (MalformedURLException e) { e.printStackTrace(); } }/** * Simply selects the home view to render by returning its name. */ @RequestMapping(value = "/characters/{characterId}", method = RequestMethod.GET) @ResponseBody public Character findCharacter(@PathVariable int characterId) { return characters.get(characterId);}}First part is a map where all characters are stored. I have used this approach to not focus in data access. Then findCharacter method that is called when URI is /characters/{characterId}. This is a URI template and is a URI-like string, containing one or more variable names, which can be accessed using @PathVariable annotation. So when you are accessing to /characters/1 parameter characterId is bound to 1. Last important part is @ResponseBody annotation. This annotation can be put on a method and indicates that the return type should be written straight to the HTTP response body, and not placed in a Model, or interpreted as a view name as standard behaviour of Spring MVC. So findCharacter method returns a Character object. And that’s all if you execute this code, and for example you enter URI http://localhost:8080/RestServer/characters/1 the output (using RestClient UI) will be:And now is when you are wondering, ¿If I am returning a Character object and output is a XML, where is conversion between object and XML? So easy, let me introduce a new concept: HttpMessageConverters. HttpMessageConverter is responsible for converting from HTTP request message to an object and converting from an object to HTTP response body. Next HttpMessageConverters are registered by default:- ByteArrayHttpMessageConverter - StringHttpMessageConverter - ResourceHttpMessageConverter - SourceHttpMessageConverter - XmlAwareHttpMessageConverter - Jaxb2RootElementHttpMessageConverter - MappingJacksonHttpMessageConverterSo now you understand why works perfectly. When you are returning Character instance, Jaxb2RootElementHttpMessageConverter using canWrite method checks if class contains XmlRootElement annotation. If class is annotated, write method is called. In this case Jaxb marshaller is called, and XML is returned. Same from XML to object but using Jaxb unmarshaller class.So easy, no complicated configurations, no complicated mappings, no unclear code, and you only need to worry about your model objects, not in conversion. But let me introduce one change. Now instead of returning XML we want to return JSON.Change could not be easier, add Jackson library to pom.xml and change @XmlRootElement to @JsonAutoDetect. And now MappingJacksonHttpMessageConverter will handle this object and will transform Character instance to JSON protocol using Jackson library. Only changing one line of code!!!And now output will be:CONCLUSIONS Of course this is a very simple application with only one operation, but it gives you an idea of how to develop Restful web services using Spring MVC. It is a matter of time of writing all your required operations using same approach that I have used with GET. Arriving at this point I think that all of us have arrived to same conclusion. Annotations are really really powerful, and Spring MVC fits perfectly for developing RESTful web services. See you next time… Download Code. Reference: Develop Restful web services using Spring MVC from our JCG partner Alex Soto at the One Jar To Rule Them All blog. Related Articles :jqGrid, REST, AJAX and Spring MVC Integration Java RESTful API integration testing Building a RESTful Web Service with Spring 3.1 and Java based Configuration, part 2 Android JSON Parsing with Gson Revisited RESTful Web Services with RESTeasy JAX-RS on Tomcat 7 – Eclipse and Maven project Spring 3 RESTful Web Services Spring MVC Development – Quick Tutorial...
google-authenticator-logo

Google Authenticator: Using It With Your Own Java Authentication Server

The Google Authenticator application for mobile devices is a very handy application that implements the TOTP algorithm (specified in RFC 6238). Using Google Authenticator you can generate time passwords that can be used to authorize users in an authentication server that shares the secret key of the requesting users. Google Authenticator is mainly used to access Google services using two-factor authentication. However, you can take advantage of Google Authenticator to generate time based password to be authenticated by a server of yours. The implementation of such a server is pretty simple in Java and you can get some inspiration getting the source code of the Google Authenticator PAM module. In this blog post, we will go through a simple implementation of the TOTP algorithm in a Java class. Generating the Secret KeyTo generate the secret key we will use a random number generator to fill up a byte array of the required size. In this case, we want: A 16 characters Base32 encoded secret key: since Base32 encoding of x bytes generate 8x/5 characters, we will use 10 bytes for the secret key.Some scratch codes (using Google’s jargon). // Allocating the buffer byte[] buffer = new byte[secretSize + numOfScratchCodes * scratchCodeSie];// Filling the buffer with random numbers. // Notice: you want to reuse the same random generator // while generating larger random number sequences. new Random().nextBytes(buffer);Now we want to extract the bytes corresponding to the secret key and encode it using the Base32 encoding. I’m using the Apache Common Codec library to get a codec implementation: // Getting the key and converting it to Base32 Base32 codec = new Base32(); byte[] secretKey = Arrays.copyOf(buffer, secretSize); byte[] bEncodedKey = codec.encode(secretKey); String encodedKey = new String(bEncodedKey);Loading the Key Into Google Authenticator You can manually load the key into Google Authenticator, or generate a QR barcode to have the application loading it from it. If you want to generate a QR barcode using Google services, you can generate the corresponding URL with a code such as this: public static String getQRBarcodeURL( String user, String host, String secret) { String format = "https://www.google.com/chart?chs=200x200&chld=M%%7C0&cht=qr&chl=otpauth://totp/%s@%s%%3Fsecret%%3D%s"; return String.format(format, user, host, secret); }Verifying a Code Now that we’ve generated the key and our users can load them into their Google Authenticator application, we need the code required to verify the generated verification codes. Here’s a Java implementation of the algorithm specified in the RFC 6238: private static boolean check_code( String secret, long code, long t) throws NoSuchAlgorithmException, InvalidKeyException { Base32 codec = new Base32(); byte[] decodedKey = codec.decode(secret);// Window is used to check codes generated in the near past. // You can use this value to tune how far you're willing to go. int window = 3; for (int i = -window; i <= window; ++i) { long hash = verify_code(decodedKey, t + i);if (hash == code) { return true; } }// The validation code is invalid. return false; }private static int verify_code( byte[] key, long t) throws NoSuchAlgorithmException, InvalidKeyException { byte[] data = new byte[8]; long value = t; for (int i = 8; i-- > 0; value >>>= 8) { data[i] = (byte) value; }SecretKeySpec signKey = new SecretKeySpec(key, "HmacSHA1"); Mac mac = Mac.getInstance("HmacSHA1"); mac.init(signKey); byte[] hash = mac.doFinal(data);int offset = hash[20 - 1] & 0xF; // We're using a long because Java hasn't got unsigned int. long truncatedHash = 0; for (int i = 0; i < 4; ++i) { truncatedHash <<= 8; // We are dealing with signed bytes: // we just keep the first byte. truncatedHash |= (hash[offset + i] & 0xFF); }truncatedHash &= 0x7FFFFFFF; truncatedHash %= 1000000;return (int) truncatedHash; }Conclusion You can now use the Google Authenticator applications and use it to generate time based passwords for your users, authenticated against your own authentication server. As you can see, the required code is pretty simple and all of the required cryptographic functions are provided by the runtime itself. The only nuisance is dealing with signed types in Java. Enjoy! Reference: Google Authenticator: Using It With Your Own Java Authentication Server from our JCG partner Enrico Crisostomo at the The Grey Blog.  Related Articles :Android Google Maps Tutorial Open alternatives to Google Maps Google ClientLogin Utility in Java JBoss 4.2.x Spring 3 JPA Hibernate Tutorial Spring MVC3 Hibernate CRUD Sample Application GWT Spring and Hibernate enter the world of Data Grids GWT 2 Spring 3 JPA 2 Hibernate 3.5 Tutorial...
java-logo

Java 8 virtual extension methods

I’ve been following the evolution of the Java 8 Lambda expressions project for a while now, and I’m really thrilled by its current state of progress. The latest “easy-to-understand” presentation I’ve found is this one: http://blogs.oracle.com/briangoetz/resource/devoxx-lang-lib-vm-co-evol.pdf Now, as an API designer, I’m particularly interested in the concept of virtual extension methods and I was wondering whether it was also considered to introduce “final” extension methods as opposed to “default” ones. For example: interface A {void a();void b() default { System.out.println("b"); };void c() final { System.out.println("c"); };}When implementing the above interface A, one…MUST also implement a() MAY implement / override b() CANNOT override c()Advantages:API designers can create convenience methods more easily without risking client code “illegally” overriding default implementations. That’s one of the main purposes of “final”. Lambda expressions wouldn’t need to be limited to pure “functional interfaces” (single-method interfaces), as a functional interface would still be “functional” if it also had any number of final extension methods. For example, the above interface A would become a functional interface, if b() was removed or if b() was made final as well. Extension methods would have more features in common with regular methods, which can be final as well. I guess for the reflection API and for the JVM, that’s a plus. The JVM is modified anyway to support extension methods. Java 8?s momentum could be used for this feature as well, i.e. now is the right time to think about thisDisadvantages:A class could inherit multiple colliding final method implementations in the case of “diamond interface inheritance“. This could lead to new compilation errors in existing code. I guess this lack of backwards compatibility is the biggest drawback.As with multiple inheritance itself, careful API designers could further improve their API’s when using final extension methods, whereas less careful API designers might break client code. But this is the case with previous usage of “final” as well, so I think final extension methods would be a very nice addition to Java 8. See the full mail and follow-up on the lambda-dev mailing list here: http://mail.openjdk.java.net/pipermail/lambda-dev/2011-December/004426.html Reference: Java 8 virtual extension methods from our JCG partner Lukas Eder at the JAVA, SQL, AND JOOQ blog. Related Articles :Java Lambda Syntax Alternatives Java SE 7, 8, 9 – Moving Java Forward Java 7 Feature Overview Manipulating Files in Java 7  GC with Automatic Resource Management in Java 7 Java 7: try-with-resources explained...
jooq-2-logo

GROUP BY ROLLUP / CUBE

Every now and then, you come across a requirement that will bring you to your SQL limits. Many of us probably give up early and calculate stuff in Java / [or your language]. Instead, it might’ve been so easy and fast to do with SQL. If you’re working with an advanced database, such as DB2, Oracle, SQL Server, Sybase SQL Anywhere, (and MySQL in this case, which supports the WITH ROLLUP clause), you can take advantage of the ROLLUP / CUBE / GROUPING SETS grouping functions. Lets have a look at my fictional salary progression compared to that of a fictional friend, who has chosen a different career path (observe the salary boost in 2011): select 'Lukas' as employee,'SoftSkills' as company,80000 as salary,2007 as yearfrom dualunion all select 'Lukas', 'SoftSkills', 80000, 2008 from dualunion all select 'Lukas', 'SmartSoft', 90000, 2009 from dualunion all select 'Lukas', 'SmartSoft', 95000, 2010 from dualunion all select 'Lukas', 'jOOQ', 200000, 2011 from dualunion all select 'Lukas', 'jOOQ', 250000, 2012 from dualunion all select 'Tom', 'SoftSkills', 89000, 2007 from dualunion all select 'Tom', 'SoftSkills', 90000, 2008 from dualunion all select 'Tom', 'SoftSkills', 91000, 2009 from dualunion all select 'Tom', 'SmartSoft', 92000, 2010 from dualunion all select 'Tom', 'SmartSoft', 93000, 2011 from dualunion all select 'Tom', 'SmartSoft', 94000, 2012 from dualNow we’re used to gathering statistics using simple grouping and simple aggregate functions. For instance, let’s calculate how much Lukas and Tom earned on average over the past few years: with data as ([above select])select employee, avg(salary)from datagroup by employeeThis will show that Lukas has earned more: +--------+-----------+ |EMPLOYEE|AVG(SALARY)| +--------+-----------+ |Lukas | 132500| |Tom | 91500| +--------+-----------+So it’s probably interesting to find out what they have earned on average in which company: with data as (...)select company, employee, avg(salary)from datagroup by company, employeeorder by company, employeeAnd immediately, it becomes clear where the big bucks are and that Tom has made a bad decision +----------+--------+-----------+ |COMPANY |EMPLOYEE|AVG(SALARY)| +----------+--------+-----------+ |jOOQ |Lukas | 225000| |SmartSoft |Lukas | 92500| |SmartSoft |Tom | 93000| |SoftSkills|Lukas | 80000| |SoftSkills|Tom | 90000| +----------+--------+-----------+ROLLUP By adding grouping fields, we “lose” some aggregation information. In the above examples, the overall average salary per employee is no longer available directly from the result. That’s obvious, considering the grouping algorithm. But in nice-looking reports, we often want to display those grouping headers as well. This is where ROLLUP, CUBE (and GROUPING SETS) come into play. Consider the following query: with data as (...)select company, employee, avg(salary)from datagroup by rollup(company), employeeThe above rollup function will now add additional rows to the grouping result set, holding useful aggregated values. In this case, when we “roll up the salaries of the company”, we will get the average of the remaining grouping fields, i.e. the average per employee: +----------+--------+-----------+ |COMPANY |EMPLOYEE|AVG(SALARY)| +----------+--------+-----------+ |SmartSoft |Tom | 93000| |SoftSkills|Tom | 90000| |{null} |Tom | 91500| |jOOQ |Lukas | 225000| |SmartSoft |Lukas | 92500| |SoftSkills|Lukas | 80000| |{null} |Lukas | 132500| +----------+--------+-----------+Note how these rows hold the same information as the ones from the first query, where we were only grouping by employee… This becomes even more interesting, when we put more grouping fields into the rollup function: with data as (...)select company, employee, avg(salary)from datagroup by rollup(employee, company)As you can see, the order of grouping fields is important in the rollup function. The result from this query now also adds the overall average salary paid to all employees in all companies +----------+--------+-----------+ |COMPANY |EMPLOYEE|AVG(SALARY)| +----------+--------+-----------+ |SmartSoft |Tom | 93000| |SoftSkills|Tom | 90000| |{null} |Tom | 91500| |jOOQ |Lukas | 225000| |SmartSoft |Lukas | 92500| |SoftSkills|Lukas | 80000| |{null} |Lukas | 132500| |{null} |{null} | 112000| +----------+--------+-----------+In order to identify the totals rows for reporting, you can use the GROUPING() function in DB2, Oracle, SQL Server and Sybase SQL Anywhere. In Oracle and SQL Server, there’s the even more useful GROUPING_ID() function: with data as (...)select grouping_id(employee, company) id, company, employee, avg(salary)from datagroup by rollup(employee, company)It documents on what “grouping level” of the rollup function the current row was produced: +----+----------+--------+-----------+ | ID|COMPANY |EMPLOYEE|AVG(SALARY)| +----+----------+--------+-----------+ | 0|SmartSoft |Tom | 93000| | 0|SoftSkills|Tom | 90000| | 1|{null} |Tom | 91500| | 0|jOOQ |Lukas | 225000| | 0|SmartSoft |Lukas | 92500| | 0|SoftSkills|Lukas | 80000| | 1|{null} |Lukas | 132500| | 3|{null} |{null} | 112000| +----+----------+--------+-----------+CUBE The cube function works similar, except that the order of cube grouping fields becomes irrelevant, as all combinations of grouping are combined. This is a bit tricky to put in words, so lets put it in action: with data as (...)select grouping_id(employee, company) id, company, employee, avg(salary)from datagroup by cube(employee, company)In the following result, you will get:GROUPING_ID() = 0: Average per company and employee. This is the normal grouping result GROUPING_ID() = 1: Average per employee GROUPING_ID() = 2: Average per company GROUPING_ID() = 3: Overall average+----+----------+--------+-----------+ | ID|COMPANY |EMPLOYEE|AVG(SALARY)| +----+----------+--------+-----------+ | 3|{null} |{null} | 112000| | 2|jOOQ |{null} | 225000| | 2|SmartSoft |{null} | 92800| | 2|SoftSkills|{null} | 86000| | 1|{null} |Tom | 91500| | 0|SmartSoft |Tom | 93000| | 0|SoftSkills|Tom | 90000| | 1|{null} |Lukas | 132500| | 0|jOOQ |Lukas | 225000| | 0|SmartSoft |Lukas | 92500| | 0|SoftSkills|Lukas | 80000| +----+----------+--------+-----------+In other words, using the CUBE() function, you will get grouping results for every possible combination of the grouping fields supplied to the CUBE() function, which results in 2^n GROUPING_ID()’s for n “cubed” grouping fields Support in jOOQ jOOQ 2.0 introduces support for these functions. If you want to translate the last select into jOOQ, you’d roughly get this Java code: // assuming that DATA is an actual table...create.select(groupingId(DATA.EMPLOYEE, DATA.COMPANY).as("id"),DATA.COMPANY, DATA.EMPLOYEE, avg(SALARY)).from(DATA).groupBy(cube(DATA.EMPLOYEE, DATA.COMPANY));With this powerful tool, you’re ready for all of those fancy reports and data overviews. For more details, read on about ROLLUP(), CUBE(), and GROUPING SETS() functions on the SQL Server documentation page, which explains it quite nicely: http://msdn.microsoft.com/en-us/library/bb522495.aspx Reference: GROUP BY ROLLUP / CUBE from our JCG partner Lukas Eder at the JAVA, SQL, AND JOOQ blog. Related Articles :Database schema navigation in Java Problems with ORMs SQL or NOSQL: That is the question? What is NoSQL ?...
oasis-saml-logo

Single Sign On for the cloud: SAML & OpenId

When accessing different applications owned by different organizations, having to authenticate everytime you go from one application to another is annoying. Not only is it time consuming, but you also have to remember multiple passwords, which are often lost. Single sign on is the ability to authenticate once and be able to move between applications seamlessly using the authenticated identity. Within an intranet or between applications that are under the control of one development organization, single sign on for web applications can be easily implemented by generating a sessionid and passing it around using cookies. However, such a solution is proprietary and will not work if you need to leave the intranet and access other applications on the cloud. To interoperate with applications on the cloud, a more standards based solution is required. A related concept and benefit is federated identity. Organizations can agree to a common name to refer to users. The user and his attributes needs to be created only in once place and others can refer to this information. In this blog, we briefly examine two popular protocols that can be used for single sign on on the cloud: SAML and OpenId. OpenId The problem OpenId solves is that you as a user do not have to maintain and provide a password to each and every site you visit. You maintain your password or other identifying credential with one provider known as the OpenId provider. The website or application that you visit and that requires proof of who you are, relies on the OpenId provider to verify that you are who you claim to be. This is known as the relying party. The basics of the OpenId protocol are: 1. You visit a web application (relying party) and enter an OpenId 2. Based on your OpenId, the relying party determines who your OpenId provider is. 3. The relying party redirects your request to the OpenId provider. 4. If you are already authenticated, this step is skipped. The OpenId provider authenticates you by asking for a password or other information. The provider warns you that the relying party is requesting information about you. 5. The request is redirected back to the relying party where you are shown the URL you were trying to access. The protocol does not require providers or relying parties to be registered anywhere. It uses plain HTTP requests and responses. The protocol messages are plain text key value pairs. The protocol works well with modern “Web20″ AJAX style applications. The OpenId protocol originated from consumer oriented websites such as Google, Twitter, Facebook etc and that is where this protocol is popular. The OpenId specification is described at OpenId specification There is a java implementation of OpenId at openid4java SAML (Security Assertion Markup Language) SAML is a XML based protocol that enables web based authentication, authorization and single sign on. SAML involves a relying party requesting an assertion and a SAML provider responding with the assertion. Examples of assertions are :Authentication Assertion : This user was authenticated using such and such method at time t. Attribute Assertion : This user has a title supermanager. Authorization Assertion : This user has permission to delete file xyz.doc.A typical SAML interaction would be as follows: 1. A user tries to access a URL or a web application which is the relying party 2. The relying party creates a SAML authentication request. 3. The relying party redirects the users browser to a SAML provider. Embedded in the request is the SAML authentication request. 4. The SAML provider evaluates the SAML request and authenticates the user. 5. The SAML provides returns a SAML authentication response to the user browser. 6. The browser forwards the SAML response back to the relying party. 7. The relying party verifies and interprets the SAML response. 8. If the response implies successful authentication, the user is redirected to the URL, he was originally trying to reach. SAML has the concept of profiles. The interaction is different based on the profile. The interaction above is the Web SSO profile. SAML has its origins more in enterprise software, in web services, in B2B communication and is from the early 2000s when XML was very popular. In fact SAML1.x had only a SOAP binding. The SAML specification is at SAML Specification There is a SAML implementation at OpenSAML Which protocol should I use ? OpenId is a simpler protocol. But has SAML has more features. OpenId supports discovery of the OpenId provider. Using SAML has generally required expensive SAML projects. OpenId supports only service provider initiated SSO. You go to a service provider web site and they need authentication. They start the conversation with the OpenId provider. SAML can also support identity provider initiated SSO. You are authenticated into your companys portal. Your company has a partner travel website for business travel. With SAML, you can go from your companys portal ( SAML provider) to the partner website ( relying party) without needing reauthentication. SAML has been around longer than OpenId. SAML is more popular in the enterprise where as OpenId is more popular in consumer oriented applications. Both OpenId and SAML rely on external transport layer security protocols such as SSL for the security of protocol messages. If you are starting a new website and want to accept users from other popular sites such as google or twitter, you might consider OpenId. However if you are an enterprise and you want your authenticated users to access your partner sites without re-authentication, you might need SAML. In summary, SAML is a feature rich protocol more popular in the enterprise. OpenId is simpler protocol with some limitations. Reference: Single Sign On for the cloud: SAML & OpenId from our JCG partner Manoj Khangaonkar at the The Khangaonkar Report blog. Related Articles :Securing GWT apps with Spring Security Developing and Testing in the Cloud Securing a RESTful Web Service with Spring Security 3.1, part 3 Apache Shiro : Application Security Made Easy Basic and Digest authentication for a RESTful Service with Spring Security 3.1, part 6 2011: The State of Software Security and Quality...
software-development-2-logo

2011: The State of Software Security and Quality

It’s the end of the year. Time to look back on what you’ve done, what you’ve learned, your successes and mistakes, and what you learned from them. I also like to look at the big picture: not just my team and the projects that I manage, or even the company that I work for, but software development in general. How are we doing as an industry, are we getting better, where are we falling behind, what are the main drivers for developers and development managers? Besides the usual analysis from InformationWeek, and from Forrester and Gartner (if you can afford them), there’s some interesting data available from software quality and software security vendors.CAST Software, a vendor of code structural analysis tools, publishes an annual Report on Application Software Health based on an analysis of 745 applications from 160 companies (365 million lines of code) using CAST’s static analysis platform. A 25-page executive summary of the report is available free with registration – registration also provides you access to some interesting research from Gartner and Forrester Research, including a pretty good paper from Forrester Research on application development metrics. The code that CAST analyzed is in different languages, approximately half of it in Java. Their findings are kinda interesting:They found no strong correlation between the size of the system and structural quality except for COBOL apps where bigger apps have more structural quality problems. COBOL apps also tended to have more high-complexity code modules. I thought that the COBOL findings were interesting, although I haven’t worked with COBOL code in a long long time. Maybe too interesting – CAST decided that the findings for COBOL were so far outside of the norm that “consequently we do not believe that COBOL applications should be directly benchmarked against other technologies”. For 204 applications, information was included on what kind of application development method was followed. Applications developed with Agile methods and Waterfall methods had similar profiles for business risk factors (robustness, performance, security) but Agile methods were not as effective when it comes to cost factors (transferability, changeability), which would seem counter-intuitive, given that Agile methods are intended to reduce the cost of change. The more releases per year, the less robust, less secure and less changeable the code was. Even the people at CAST don’t believe this finding.CAST also attempts to calculate an average technical debt cost for applications, using the following formula: (10% of low severity findings + 25% of medium severity findings + 50% of high severity findings) * # of hours to fix a problem * cost per hour for development time The idea is that not all findings from the static analysis tool need to be fixed (maybe 10% of low severity findings and 25% of medium severity findings), but at least half of the high severity issues found need to be fixed. They assume that on average each of these fixes can be made in 1 hour, and estimate that the average development cost for doing this work is $75 per hour. This results in an average technical debt cost of $3.61 per LOC. For Java apps, the cost is much higher, at $5.42 per LOC. So your average 100 KLOC Java system carries about $500,000 worth of technical debt…. It’s difficult to say how real or meaningful these findings are. The analysis depends a lot on the extensiveness of CAST’s static analysis checkers (with different checking done for different languages, making it difficult to compare findings across languages) and the limited size of their customer base. As their tools improve and their customer base grows, this analysis will become more interesting, more accurate and more useful. The same goes for the State of Software Security Report from Veracode, a company that supplies static and dynamic software security testing services. Like the CAST study, this report attempts to draw wide-ranging conclusions from a limited data set – in this case, the analysis of almost 10,000 “application builds” over the last 18 months (which is a lot less than 10,000 applications, as the same application may be analyzed at least twice if not more in this time window). The analysis focused on web apps (75% of the applications reviewed were web apps). Approximately half of the code was in Java, one quarter in .NET, the rest in C/C++, PHP, etc. Their key findings:8 out of 10 apps fail to pass Veracode’s security tests on the first pass – the app contains at least 1 high-risk vulnerability. For web apps, the top vulnerability is still XSS. More than half of the vulnerabilities found are XSS, and 68% of web apps are vulnerable to XSS attacks. 32% of web apps were vulnerable to SQL Injection, even though only 5% of all vulnerabilities found were SQL Injection issues. For other apps, the most common problems were in error handling (19% of all issues found) and cryptographic mistakes (more than 46% of these apps had issues with crypto).Another interesting analysis is available from SmartBear Software, a vendor of software quality tools, which recently sponsored a webinar by Capers Jones on The State of Software Quality in 2011. Capers Jones draws from a much bigger database of analysis over a much longer period of time. He provides a whirlwind tour of facts and findings in this webinar, summarizing and updating some of the material that you can find in his books. SmartBear provides some other useful free resources, including a good white paper on best practices for code review. This is further explored in the chapter on Modern Code Reviews by SmartBear’s Jason Cohen in the book Making Software: What Really Works, and Why we Believe it, a good collection of essays on software development practices and the state of research in software engineering. All of these reports are free – you need to sign up, but you will get minimal hassle – at least I haven’t been hassled much so far. I’m not a customer of any of these companies (we use competing or alternative solutions that we are happy with for now) but I am impressed with the work that these companies do to make this information available to the community. There were no real surprises from this analysis. We already know what it takes to build good software, we just have to make sure that we actually do it.Reference: 2011: The State of Software Security and Quality from our JCG partner Jim Bird at the Building Real Software blog. Related Articles :Not doing Code Reviews? What’s your excuse? Fix That Code Immediately! The Pragmatic Programmer – Review / Summary Notes. Principles for Creating Maintainable and Evolvable Tests The Three Ways to Work with Code Diminishing Returns in software development and maintenance...
software-development-2-logo

People Related Classic Mistakes

In my last blog I mentioned Rapid Development: Taming Wild Software Schedules by Steve McConnell, which although has now been around for at least 10 years is still as relevant today as when it was written. One of my favourite parts of the book was his treatment of Classic Mistakes As Steve’s book states, classic mistakes are classic mistakes because they’re mistakes that are made so often and by so many people. They have predictably bad results and, when you know them, they stick out like a sore thumb. Half the battle in shipping quality software is avoiding making mistakes – something that is easier said than done. The other half of the battle is using efficient development practices that ensure well written, thought-out, designed and tested code. The idea behind listing them here is that, once you know them, you can spot them and hopefully do something to remedy their effect. Classic mistakes can be divided in to four types:People Related Mistakes Process Related Mistakes Product Related Mistakes Technology Related MistakesToday’s blog takes a quick look at the first of Steve’s categories of mistakes: People Related Mistakes. Undermined Motivation Avoid doing things that undermine the motivation of other members of your team. Weak Personnel Hire the best people you can find. Never hire from the bottom of the barrel. Uncontrolled Problem Employees Not sacking problem employees results in lower motivation for the rest of the team and the reworking of their areas of code. Heroics Allowing heroic gung-ho developers is similar to putting a bull in a china shop. Gung-ho heroics threaten the schedule because slips aren’t detected until its too late. Some managers praise heroic developers working all hours battling away to get a project completed on time. Isn’t it better to have steady, consistent 9 to 5 working and progress reporting. Get a life. If you have to work heroically then project is in big trouble. Adding People to a Project Late Adding people to a project that is behind schedule does more damage than doing nothing. Noisy Crowded Offices Workers who occupy quiet, private offices work better than those who occupy noisy open plan offices or cubicles. Friction Between Developers and Customers There is often friction between developers and their customers to the extent that it is not uncommon for both parties to consider cancelling the project. Friction is primarily caused by poor communications leading to poorly understood requirements and poor user interfaces. In the worst cases the client can refuse to accept the product. Unrealistic Expectations Here managers or clients expect that something can be done within a certain time scale – usually with no sound basis. This doesn’t lengthen time scales as such but gives the perception of missed deadlines and schedules that are too long. Software development takes as long as it takes. Lack of Effective Project Sponsorship Every project should have some one high enough, or capable of saying NO when some other high level manager tries to force you to agree unrealistic time-scales or make changes that undermine the project. This is important as lack of project sponsorship virtually guarantees failure. A high level sponsor should support aspects such as realistic planning, change control, practices etc. Lack of Stake Holder Buy-in Close co-operation only occurs when all major players in the project, from the top to the bottom put their intellect and personal force behind a project. Team work pays off. Lack of User Input Projects without early end user input risk allowing misunderstandings to occur in the requirements. This leads to time consuming feature creep later. Politics Placed Over Substance A team (or manager) that puts politics over results will tend to be well regarded for about six months. After that someone generally sees through them with fatal consequences. Wishful Thinking Wishful thinking isn’t just optimism. It’s closing your eyes and hoping something works when you have no reasonable basis for thinking it. It’s knowing that something can’t be done but agreeing to try anyway. Wishful thinking at the beginning of a project leads to big blowups at the end. It undermines planning and is at the root of more problems than all the other mistakes combined. Reference: People Related Classic Mistakes from our JCG partner Roger Hughes  at the Captain Debug’s Blog . Related Articles :Top 10 Java Books you don’t want to miss Java Developer Most Useful Books Book Review: The Passionate Programmer written by Chad Fowler The Pragmatic Programmer – Review / Summary Notes Programming Android Online Book Book Review: The Passionate Programmer written by Chad Fowler The top 9+7 things every programmer or architect should know Things Every Programmer Should Know...
software-development-2-logo

Transactional event-based NOSQL storage

I am presenting here a simple two steps architectural approach based on stored events as a workaround for the lack of full atomic transaction support in so-called “NOSQL” databases. Being fairly new to NOSQL-based architectures, I have the annoying intuition that I am about to write nothing but a set of obvious statements. On the other hand, I have not yet read a detailed description of this anywhere, so hopefully it will be useful to some other developers as well.Problem statement NOSQL databases do not offer atomic transactions over several update operations on different data entities(*). A simplistic explanation to this is that the two-phase commit mechanism forces each participant to lock its data for some time, which does not scale with highly partitioned data (cf CAP theorem). However, NOSQL databases of type document stores are usually able to update atomically as many properties of one single document as we want (this is at least true for AWS SDB data rows and MongoDB JSON documents).* [UPDATE 10 Sept 2011] : I recently discovered the awesome neo4j graph database that brands itself as “NOSQL” and offers support for traditional ACID transactions. However, even though neo4j can be deployed as a cluster in replication mode, it is is not partition tolerant, as explained here. Redis is also offering some interesting atomic transactional support, although, as pointed out here, a failed update operation does not trigger the automatic roll back of the others.Data consistency resulting from a partial update failuresThere exists of course tons of situations which require updating several entities as one single operation. In the NOSQL world, there’s even more such examples as we are encouraged to replicate data across entities rather that use joins at read time, for performance reasons(*).* [UPDATE 10 Sept 2011] : Those operations are costly in most current NOSQL products because of the so-called n+1 problem. When using a RDBMS solution, this is typically resolved by letting the DB itself perform the join internally before returning data to the application. I learnt recently that some NOSQL products also allow the traversal of relationships inside the DB: neo4j is of course one of them, by nature, but Redis also provides such mechanism, as explained at the end of this post, and here. A naïve solution to this is to ignore the lack of atomicity and simply leave the data inconsistent in case of partial update failure. This might be acceptable when the data is considered less critical than the throughput (e.g collection of statistical trends) or in situations where inconsistency just means we end up with some non deleted “dead” records that will be ignored forever. Another approach is trying to compensate by undoing the already executed operations in case of failure. However, obtaining 100% guarantee of consistency that way is hard because it requires to build an undo mechanism that never fails, which can be a very hard thing to do.A brief example Imagine an application that allow users to create a profile and to send messages to each other. Imagine the requirements tell us that, when displaying the list of messages, we must display the city of origin of both the author and the recipient. A relational model for this would simply use foreign keys to link the entities.RDB model for the user profile exampleIn the NOSQL world however, this is discouraged because traversing the relations requires supplementary round-trips to the DB (I admit the performance impact would be small in the case of my simplistic example). We therefore choose to de-normalize the data and to replicate the personal details of the author and recipient within the userMessage table. This of course yields the need of several update operations any time a profile is updated (because the messages must now be updated as well). In a lot of applications, read operations happen much more often then write operations, so the overall performance of this approach is typically good.NOSQL model for the user profile exampleSolution overview I just found that Adam Heroku has already blogged about a very similar solution 4 years ago (argl !), here: http://adam.heroku.com/past/2007/12/17/a_world_without_sql/ “The old example of using a database transaction to wrap the transfer of money from one bank account to another is total bull. The correct solution is to store a list of ledger events (transfers between accounts) and show the current balance as a sum of the ledger. If you’re programming in a functional language (or thinking that way), this is obvious.” Which seems to confirm I am just stating the obvious hereCourageously going on anyway… The solution proposed here is using two distinct data spaces, one for writing the data and one for reading it. Behind the scene, a job is asynchronously pulling data out of the first space and propagating its impact to the second one. Both spaces have very different data models. In this post, I refer to those spaces as the write domain and the read domain.Writing events As explained in the problem statement above, the largest set of data we can update in one atomic operation is one single “row”. For each desired update transaction, the “write-only DAOs” will therefore output nothing more than one monolithic event describing in details the business action(s) that has just happened. In my example, those events would be instances of “ProfileUpdateEvent”. In the bank account example of Heroku, these would be the “ledgers”. The events must be self-contained, i.e. bundle enough information so that their impact on the domain entities can later be completely evaluated based on their content only. During this later evaluation, they will be seen as snapshots from the past, possibly hard to correlate with the present state of other resources. It is also important to fully validate the business action before outputting the resulting event to DB. The propagation job runs asynchronously and will have no simple way of warning the initiator of the update that some input parameters is incorrect or inconsistent.A nice improvement is to try to trigger the propagation job described in the next section directly after the event insertion in DB. If this second step fails, it’s still ok as we know it is going to be retried later on anyway. We can even put in place a nice degradation of service policy which only triggers this immediate propagation when the system load is below a given threshold. This ensures that we free up computing resources (which are infinite in a cloud, I know, I just do not trust this ^__^ ) in case of high traffic.Propagation job: The propagation job is a process which runs continuously in the background, its role is fairly straightforward:Read one event out from the write domain (or several correlated events, if this is relevant) Update the corresponding read domain tables in an idempotent fashion If all the above was successful, remove the event from the write domainThis propagation of course involves several update operations wich, here again, are not guaranteed to be executed atomically. However, because the events are supposed to be fully validated at previous step, any failure here should only happen in case of recoverable error (like a network outage, a node crash…), which implies that if we retry the failed propagation regularly, it should eventually succeed. If this is not the case, it means we have a bug, which requires a human intervention anyway. It is therefore crucially important that the update operation performed on the read domain be idempotent, so that we can repeat the complete set of operations (even the ones that did not fail last time). Idempotency is easily achieved in the user profile example: we can execute the “setAuthorCity” method repeatly with the same argument with no side effect. It is trickier to achieve in the ledger example of Heroku but still feasible, for example by recording the list of “received event ids” as part of the updated row (it must be within the updated row itself, here again because this is the biggest atomic write operation we can execute).Reading data The read operations boil down to, well, reading the data from the read domain any time there is a business need to do so. A supplementary step can improve the consistency of the obtained information by reading all pending events related to the entities obtained from the read domain and computing in real time the freshest possible state of the domain entities. Similarly to the degradation of service mechanism exposed in the “Writing events” section above, in case of peak of application traffic, the read DAOs could be instructed not to perform this additional step in order to return a slightly less consistent result but also to free resources faster.Conclusion The event-based storage solution proposed here lets a system benefit from the high scalability offered by NOSQL databases while still providing a 100% guarantee of “eventual consistency” for any written data, even when several update operations need to be executed as one business transactional unit. The approach does not prevent us to define a very de-normalized data model, which avoid join operations and increase read access speed. Three important aspects of the solution areevents must be fully validated before being put in DB events must be self contained propagation of the impact must be done in an idempotent wayBoth read and write operations are much simpler than in traditional approaches as the data model they handle is well aligned with their respective use cases and need little manipulation when moved to/from the persistence layer. The business logic is cleanly encapsulated in the propagation job(s), away from the application external interfaces, so the approach also helps us enforce good architectural practices. Because the propagation job can optionally be triggered by the write operations immediately, outputting the event only appears as one small supplementary step that ensures eventual consistency of data. The required latency before a written data becomes visible to the read-only DAOs should therefore not be, under normal traffic, higher than the latency we already experience in NOSQL systems. Reference: Transactional event-based NOSQL storage from our JCG partner Svend Vanderveken  at the Svend blog. Related Articles :What is NoSQL ? SQL or NOSQL: That is the question? Developing and Testing in the Cloud Spring Declarative Transactions Example The Architecture of Open Source Applications Erlang vs Java memory architecture...
jcg-logo

Best Of The Week – 2011 – W51

Hello guys, Time for the “Best Of The Week” links for the week that just passed. Here are some links that drew Java Code Geeks attention: * Spring 3.1 is GA Today – Adds Java 7 Support, Environment Abstraction and Cache Abstraction:Following the announcement for the general availability of Spring 3.1, this article provides some information  on the new releases and the major new features for Spring 3.1, which include Java 7 support, environment abstraction and cache abstraction. * Estimating Java Object Sizes with Instrumentation: This tutorial describes how to use the Instrumentation interface in order to get an estimate of  the size of a particular object via its getObjectSize(Object) method. * More Android Games that Play Nice:A post with some tips on Android game development, like how to handle the press of the HOME and BACK buttons and how to get control of the audio in the standard and proper way. Also  check out some Android OpenGL ES tutorials. * Making the Mobile Web Native with PhoneGap: This article presents PhoneGap for developing cross-platform mobile applications for iOS and Android.  PhoneGap is an HTML5 app platform that allows you to  author native applications with web technologies and get  access to APIs and app stores. * The Top Java Memory Problems – Part 2: A must read article for every Java developer, identifies the top Java memory problems including  high memory usage, using the HTTP Session as Cache, using caches in a wrong way, creating  large temporary objects, loading the same class in memory multiple times etc. Also see How to get C like performance in Java and Low GC in Java: Use primitives instead of wrappers. * Top Ten Reasons Why Large Companies Fail To Keep Their Best Talent: Nice article listing the top reasons that companies fail to keep their best people around. Reasons  include bureaucracy, boring projects to work with, no discussion about career development,  lack of open-mindedness etc. * Using JSON to Build Efficient Applications: This article is a nice introduction to JSON, a way to store information in an organized, easy-to-access way. It also provides a number of reasons on why use JSON (e.g. it is lightweight, integrates easily with every programming language, is natively supported in Javascript etc.). Also take a look at Java JSON processing with Jackson, Add JSON capabilities into your GWT application and Android JSON Parsing with Gson Tutorial. * Capture Stalls in Your Java Application With jHiccup: This article presents jHiccup, a performance monitoring tool that measures any hiccups (pauses or stalls). Addtionally  it can identify whether the cause of a stall is in your application code or the underlying runtime platform. * Spring Web and SmartGWT integration: A nice tutorial that shows how to integrate Spring Web and SmartGWT with JSON over HTTP. Also  check out Getting Started with SmartGWT for awesome GWT interfaces and Spring MVC Development – Quick Tutorial. * Top 5 enhancements of Spring MVC 3.1: This article presents some of the new features of Spring MVC 3.1, including flash attributes support (used to store attributes between requests ), support for JSR-303 validation groups, support for @Valid on @RequestBody method arguments, support for PUT request with form encoded data and finally Java based configuration. * Android 101: How to monetize your Android application: An article discussing in short how to monetize an Android application and provides some  tips on Paid apps, In-app purchase and using Ads. Also check out our “Android Full Application Tutorial” series to learn how to build an Android application. That’s all for this week. Stay tuned for more, here at Java Code Geeks. Cheers, Ilias Related Articles:Best Of The Week – 2011 – W50 Best Of The Week – 2011 – W49 Best Of The Week – 2011 – W48 Best Of The Week – 2011 – W47 Best Of The Week – 2011 – W46 Best Of The Week – 2011 – W45 Best Of The Week – 2011 – W44 Best Of The Week – 2011 – W43 Best Of The Week – 2011 – W42 Best Of The Week – 2011 – W41...
java-logo

Using a ThreadPoolExecutor to Parallelize Independent Single-Threaded Tasks

The task execution framework, introduced in Java SE 5.0, is a giant leap forward to simplify the design and the development of multi threaded applications. The framework provides facilities to manage the concept of task, to manage thread life cycles and their execution policy. In this blog post we’ll describe the power, the flexibility and the simplicity of this framework showing off a simple use case. The Basics The executor framework introduces an interface to manage task execution: Executor. Executor is the interface you use to submit tasks, represented as Runnable instances. This interface also isolates a task submission from a task execution: executors with different execution policies all publish the same submission interface: should you change your execution policy, your submission logic wouldn’t be affected by the change. If you want to submit a Runnable instance for execution, it’s as simple as: Executor exec = …; exec.execute(runnable);Thread Pools As outlined in the previous section, how the executor is going to execute your runnable isn’t specified by the Executor contract: it depends on the specific type of executor you’re using. The framework provides some different types of executors, each one with a specific execution policy tailored for different use cases. The most common type of executors you’ll be dealing with are thread pool executors., which are instances of the ThreadPoolExecutor class (and its subclasses). Thread pool executors manage a thread pool, that is the pool of worker threads that’s going to execute the tasks, and a work queue. You surely have seen the concept of pool in other technologies. The primary advantage of using a pool is reducing the overhead of resources creation, reusing structures (in this case, threads) that have been released after use. Another implicit advantage of using a pool is the capability of sizing your resource usage: you can tune the thread pool sizes to achieve the load you desire, without jeopardizing system resources. The framework provides a factory class for thread pools called Executors. Using this factory you’ll be able to create thread pools of different characteristics. Often, the underlying implementation is often the same (ThreadPoolExecutor) but the factory class helps you quickly configure a thread pool without using its more complex constructor. The factory methods are:newFixedThreadPool: this method returns a thread pool whose maximum size is fixed. It will create new threads as needed up to the maximum configured size. When the number of threads hits the maximum, the thread pool will maintain the size constant. newCachedThreadPool: this method returns an unbounded thread pool, that is a thread pool without a maximum size. However, this kind of thread pool will tear down unused thread when the load reduces. newSingleThreadedExecutor: this method returns an executor that guarantees that tasks will be executed in a single thread. newScheduledThreadPool: this method returns a fixed size thread pool that supports delayed and timed task execution.This is just the beginning. Executors also provide other facilities that are out of scope in this tutorial and that I strongly encourage you to study about:Life cycle management methods, declared by the ExecutorService interface (such as shutdown() and awaitTermination()). Completion services to poll for a task status and retrieve its return value, if applicable.The ExecutorService interface is particularly important since it provides a way to shutdown a thread pool, which is something you almost surely want to be able to do cleanly. Fortunately, the ExecutorService interface is pretty simple and self-explanatory and I recommend you study its JavaDoc thoroughly. Basically, you send a shutdown() message to an ExecutorService, after which it won’t accept new submitted tasks, but will continue processing the already enqueued jobs. You can pool for an executor service’s termination status with isTerminated(), or wait until termination using the awaitTermination(…) method. The awaitTermination method won’t wait forever, though: you’ll have to pass the maximum wait timeout as a parameter. Warning: a source of errors and confusion is a understanding why a JVM process never exits. If you don’t shutdown your executor services, thus tearing down the underlying threads, the JVM will never exit: a JVM exits when its last non-daemon thread exits. Configuring a ThreadPoolExecutor If you decide to create a ThreadPoolExecutor manually instead of using the Executors factory class, you will need to create and configure one using one of its constructors. The most extensive constructor of this class is: public ThreadPoolExecutor( int corePoolSize, int maxPoolSize, long keepAlive, TimeUnit unit, BlockingQueue<Runnable> workQueue, RejectedExecutionHandler handler);As you can see, you can configure:The core pool size (the size the thread pool will try to stick with). The maximum pool size. The keep alive time, which is a time after which an idle thread is eligible for being torn down. The work queue to hold tasks awaiting execution. The policy to apply when a task submission is rejected.Limiting the Number of Queued Tasks Limiting the number of concurrent tasks being executing, sizing your thread pool, represents a huge benefit for your application and its execution environment in terms of predictability and stability: an unbounded thread creation will eventually exhaust the runtime resources and your application might experience as a consequence, serious performance problems that may lead even to application instability. That’s a solution to just one part of the problem: you’re capping the number of tasks being executed but aren’t capping the number of jobs that can be submitted and enqueued for later execution. The application will experience resource shortage later, but it will eventually experience it if the submission rate consistently outgrows the execution rate. The solution to this problem is:Providing a blocking queue to the executor to hold the awaiting tasks. In the case the queue fills up, the submitted task will be “rejected”. The RejectedExecutionHandler is invoked when a task submission is rejected, and that’s why the verb rejected was quoted in the previous item. You can implement you’re own rejection policy or use one of the built-in policies provided by the framework.The default rejection policies has the executor throw a RejectedExecutionException. However, other built-in policies let you:Discard a job silently. Discard the oldest job and try to resubmit the last one. Execute the rejected task on the caller’s thread.When and why would one use such a thread pool configuration? Let’s see an example. An Example: Parallelizing Independent Single-Threaded Tasks Recently, I was called to solve a problem with an old job my client was running since a long time ago. Basically, the job is made up of a component that awaits for file system events on a set of directory hierarchies. Whenever an event is fired, a file must be processed. The file processing is performed by a proprietary single threaded process. Truth be said, by its own nature, even if I could, I don’t if I could parallelize it. The arrival rate of events is very high throughout part of the day and there’s no need to process file in real time, they just to get processed before the next day. The current implementation was a mix and match of technologies, including a UNIX shell script that was responsible for scanning huge directory hierarchies to detect where changes were applied. When that implementation was put in place, the number of cores in the execution environment were two, as much. Also, the rate of events was pretty lower: nowadays they’re in the order of the millions, for a total of between 1 and 2 terabytes of raw data to be processed. The servers the client is running these processes nowadays are twelve core machines: a huge opportunity to parallelize those old single-threaded tasks. We’ve got basically all of the ingredients for the recipe, we just need to decide how to build and tune it. Some thoughts before writing any code were necessary to understand the nature of the load and these are the constraints I detected:A really huge number of files is to be scanned periodically: each directory contains between one and two millions of files. The scanning algorithm is very quick and can be parallelized. Processing a file will take at least 1 second, with spikes of even 2 or 3 seconds. When processing a file, there is no other bottleneck than CPU. CPU usage must be tunable, in order to use a different load profile depending on the time of the day.I’ll thus need a thread pool whose size is determined by the load profile active at the moment of invoking the process. I’m inclined to create, then, a fixed size thread pool executor configured according to the load policy. Since a processing thread is only CPU-bound, its core usage is 100% and waits on no other resources, the load policy is very easy to calculate: just take the number of core available in the processing environment and scale it down using the load factor that’s active at that moment (and check that at least one core is used in the moment of peak): int cpus = Runtime.getRuntime().availableProcessors(); int maxThreads = cpus * scaleFactor; maxThreads = (maxThreads > 0 ? maxThreads : 1);Then, I need to create a ThreadPoolExecutor using a blocking queue to bound the number of submitted tasks. Why? Well: the directory scanning algorithms are very quick and will generate a huge number of files to process very quickly. How huge? It’s hard to predict and its variability is pretty high. I’m not going to let the internal queue of my executor fill up indiscriminately with the objects representing my tasks (which include a pretty huge file descriptor). I’ll prefer let the executor reject the files when the queue fills up. Also, I’ll use the ThreadPoolExecutor.CallerRunsPolicy as rejection policy. Why? Well, because when the queue is filled up and while the threads in the pools are busy processing the file, I’ll have the thread that is submitting the task executing it. This way, the scanning stops to process a file and will resume scanning as soon as it finishes executing the current task. Here’s the code that creates the executor: ExecutorService executorService = new ThreadPoolExecutor( maxThreads, // core thread pool size maxThreads, // maximum thread pool size 1, // time to wait before resizing pool TimeUnit.MINUTES, new ArrayBlockingQueue<Runnable>(maxThreads, true), new ThreadPoolExecutor.CallerRunsPolicy());The skeleton of the code is the following (greatly simplified): // scanning loop: fake scanning while (!dirsToProcess.isEmpty()) { File currentDir = dirsToProcess.pop();// listing children File[] children = currentDir.listFiles();// processing children for (final File currentFile : children) { // if it's a directory, defer processing if (currentFile.isDirectory()) { dirsToProcess.add(currentFile); continue; }executorService.submit(new Runnable() { @Override public void run() { try { // if it's a file, process it new ConvertTask(currentFile).perform(); } catch (Exception ex) { // error management logic } } }); }// ... // wait for all of the executor threads to finish executorService.shutdown(); try { if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) { // pool didn't terminate after the first try executorService.shutdownNow(); }if (!executorService.awaitTermination(60, TimeUnit.SECONDS)) { // pool didn't terminate after the second try } } catch (InterruptedException ex) { executorService.shutdownNow(); Thread.currentThread().interrupt(); }Conclusion As you can see, the Java concurrency API is very easy to use, very flexible and extremely powerful. Some years ago, I would have taken much more effort to write such a simple program. This way, I could quickly solve a scalability problem caused by a legacy single threaded component in a matter of hours. Reference: Using a ThreadPoolExecutor to Parallelize Independent Single-Threaded Tasks from our JCG partner Enrico Crisostomo at the The Grey Blog. Related Snippets :Blocking Queue example of limited connection pool CountDownLatch example of a more general wait/notify mechanism  Reentrant Lock example of a task runner emaphores example limiting URL connections Related Articles :Java Concurrency Tutorial – Thread Pools Beneficial CountDownLatch and tricky java deadlock Concurrency optimization – Reduce lock granularity Java Concurrency Tutorial – CountDownLatch How does JVM handle locks Java Tutorials and Android Tutorials list...
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books