Featured FREE Whitepapers

What's New Here?

apache-maven-logo

TestNG and Maven configuration guide

To be useful automatic tests should run very fast. Otherwise will not be run often during development or even will be ignored in the default configuration on the developer workstations. The simplest rule it to write only small unit tests which test given class having the neighborhood mocked. Nevertheless sometimes it is useful/required to test something in IoC container context (Spring, CDI, Guice) or using embedded database (H2, HyperSQL, Derby). Unfortunately even a few tests of that kind can increase significantly overall test execution time. I had that situation in one of my projects and to prevent using skipTests flag I developed a solution using groups from TestNG and Maven Surefire plugin. Tests were divided into three groups:very fast real unit tests (all by default) – should be run very often during development (from IDE or by mvn test, mvn package) slower integration but self-sufficient tests (setting up Spring context and/or using embedded H2 database) – should be run at least before commit/push or while working on given part (from IDE or by mvn integration-test, mvn install) real integration tests (requiring access to remote servers – e.g. testing web services or REST) – should be run daily by CI server or by developers working on the integration (mvn install, mvn integration-test with additional profile enabled)To achieve that given tests (or test classes) have to be marked as “self-integration” or “integration” (at method or class level): @Test(groups = "self-integration") public void shouldInitializeChainedAppInfoProperly() {@Test(groups = "integration") public class FancyWebServiceIntegrationTest {Maven Surefire plugin should be configured to exclude “self-integration” and “integration” test groups from default execution and add “self-integration” to “integration-test phase: <build><plugins>(...)<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>${ver.surefire-plugin}</version><executions><execution><id>default-test</id> <!-- to override default configuration - in fact: unit tests --><configuration><excludedGroups>self-integration,integration</excludedGroups></configuration></execution><execution><id>self-integration</id><phase>integration-test</phase><goals><goal>test</goal></goals><configuration><groups>self-integration</groups><reportsDirectory>target/self-integration-surefire-reports/</reportsDirectory></configuration></execution></executions></plugin></plugins></build>In addition (if needed) the separate separate profile with “integration” test group configured in “integration-test” phase could be created. <profiles>(...)<profile><id>integration</id><build><plugins><plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-surefire-plugin</artifactId><version>${ver.surefire-plugin}</version><executions><execution><id>integration</id><phase>integration-test</phase><goals><goal>test</goal></goals><configuration><groups>integration</groups><reportsDirectory>target/integration-surefire-reports/</reportsDirectory></configuration></execution></executions></plugin></plugins></build></profile></profiles>Working example can be find in AppInfo‘s artificial branch (pom.xml and sample test class). It’s easy to adopt it to your needs. All three test groups have separate reports format to not override each other. As the extension it could probably possible to merge they into one aggregated test report. Reference: Run fast unit tests often, slow integration rarely – TestNG and Maven configuration guide from our JCG partner Marcin Zajaczkowski at the Solid Soft blog. ...
play-framework-logo

Writing modules for Play 2, part 1: Get something working

A couple of weeks ago, I migrated the Play! framework 1.x version of Deadbolt to the Play 2 platform, and was surprised at the lack of information on creating modules. This topic was covered in detail in the Play 1.x docs, and this made creating modules very simple. Clearly, something needed to be done – and so this is the first in a three-part series on creating modules and plugins for Play 2. It’s important to note this is not a definitive guide – instead, it’s a collection of the information and techniques I’ve used when working on my own modules. I’m going to assume you already have a working installation of Play 2 – if you don’t, head over to http://www.playframework.org/documentation/2.0/Installing for details on how to set this up. In this first part, we’re going to cover the fundamentals of creating and publishing the module, and adding a sample application. The next two parts will go into greater depth on plugins, interceptors, tags and other useful tools. First, a small note on the difference between a module and a library. In Play 1.x, a module was created using “play new-module” and distributed via the module repository. Up until Play 1.1, modules were controlled through application.conf entries, and libraries were added locally. From Play 1.2, both modules and libraries were controlled through the dependency management mechanism based on Ivy. In both cases, though, there was a clear concept of module (something tightly integrated with Play, which followed its conventions on package structure, etc) and a library (a general third-party library). In Play 2, the line is blurred to some degree. Modules are now distributed in the same way as libraries, though Ivy or Maven, and the package structure can be arbitrary so you can have a traditional com.example.whatever structure. From this point of view, the only real difference between modules and libraries is that modules use the Play API directly. Secondly, a note on language. Since both Java and Scala are natively supported by Play 2, you can implement your module in either language. If the module has its own API that applications can use, the superb interoperability between Java and Scala means your language choice is – in most cases – irrelevant to whichever application uses your API. 1. Getting started As a simple introduction, we’re going to create a basic logging module that – following industry best practices – writes output to the console window. This module is called mylogger, because it’s Monday and I’m not feeling very creative right now. This module is going to be (mainly) written in Java. You can think of a Play 2 module as being a Play 2 application with a couple of files missing. So, you create a module in the same way as you would an application. Go to, or create, a directory where you keep your projects, and use “play new mylogger” to create the app. Choose option 2 when prompted, to create a simple Java app. steve@hex:/tmp$ play new mylogger _ _ _ __ | | __ _ _ _| | | '_ \| |/ _' | || |_| | __/|_|\____|\__ (_) |_| |__/play! 2.0, http://www.playframework.orgThe new application will be created in /tmp/myloggerWhat is the application name? > myloggerWhich template do you want to use for this new application?1 - Create a simple Scala application 2 - Create a simple Java application 3 - Create an empty project> 2OK, application mylogger is created.Have fun! Because we’re going to have a sample application alongside the module, we’re going to change the directory structure a little. At the moment, it looks like this: mylogger – app – conf – project – public – target – .gitignore – READMEIn the mylogger directory, create two new directories, project-code and samples. Copy all the files listed above into the project-code directory. You should now have mylogger – samples – project-code – app – conf – project – public – target – .gitignore – READMEThe conf directory contains two files – routes, and application.conf. – application.conf needs to be present for Play to recognise mylogger/project-code as a Play application, so we can’t delete it but we can remove everything it contains. Any configuration your module needs should be added to the application.conf of the “real” application. – routes *must* be deleted. If you don’t do this, it may/will supercede the routes file of whichever application uses it, and this is A Bad Thing (mainly because nothing will work). With application.conf emptied and routes deleted, type “play” in the project-code to start the Play console. steve@hex:/tmp/mylogger/project-code$ play [info] Loading project definition from /tmp/mylogger/project-code/project [info] Set current project to mylogger (in build file://tmp/mylogger/project-code/) _ _ _ __ | | __ _ _ _| | | '_ \| |/ _' | || |_| | __/|_|\____|\__ (_) |_| |__/play! 2.0, http://www.playframework.org> Type "help play" or "license" for more information. > Type "exit" or use Ctrl+D to leave this console.[mylogger] $We now have a valid Play 2 module (that, admittedly, doesn’t do anything). If you use an IDE, this is a good time to create the project – this tutorial is IDE-agnostic, however, so you can use the guide at http://www.playframework.org/documentation/2.0/IDE if you want to do this. 2. Adding some functionality As I mentioned above, Play 2 modules can have a more traditional package structure of com.example.whatever, but a default Play application has the tradtional app/controllers, app/models, etc, structure. We’ll keep this for now, and change it in later parts of this tutorial. 2.1 Slash and burn In the app folder, we have the following structure as created for us by the “play new” command: app – controllers – Application.java – views – index.scala.html – main.scala.htmlFor the initial iteration of this module, we don’t need any views so you can delete the views package. You can also delete Application.java, as we’re writing this from scratch. 2.2 Add some module code In the controllers package, create a new class called MyLogger.java. It doesn’t need to extend or implement anything, and it contains a single method: package controllers;/** * @author Steve Chaloner */ public class MyLogger { public static void log(String message) { System.out.println("MyLogger: " + message); } }2.3 Have a beer You’ve just written a module. Go have a beer. 2.4 Post-beer realisation As you gaze into your now-empty glass, abuzz with the joy of creation and impending feeling of industry-wide fame, you might realize that no-one can actually get to your module because it’s sitting on your computer. You need to publish it. 3. Publish and be damned For this example, we’re just going to publish to your local repository. In the root of your Play installation, there is a repository directory, and this is where you wil first push your module. Before publishing, always ensure you have run a “clean”, otherwise some classes/files that you have removed from the source tree may still exist in compiled form and end up in your module jar file. If the class matches one that’s in your actual application, it can be called in place of your actual class. Which sucks. In the Play console, use “clean” and then “publish-local” to package up your module and publish it to the local repo: [mylogger] $ publish-local [info] Updating {file:/tmp/mylogger/project-code/}mylogger... [info] Packaging /tmp/mylogger/project-code/target/scala-2.9.1/mylogger_2.9.1-1.0-SNAPSHOT-sources.jar ... [info] Done packaging. [info] Wrote /tmp/mylogger/project-code/target/scala-2.9.1/mylogger_2.9.1-1.0-SNAPSHOT.pom [info] Done updating. [info] :: delivering :: mylogger#mylogger_2.9.1;1.0-SNAPSHOT :: 1.0-SNAPSHOT :: release :: Mon Mar 19 20:57:26 CET 2012 [info] delivering ivy file to /tmp/mylogger/project-code/target/scala-2.9.1/ivy-1.0-SNAPSHOT.xml [info] Compiling 1 Java source to /tmp/mylogger/project-code/target/scala-2.9.1/classes... [info] Generating API documentation for main sources... model contains 4 documentable templates [info] Packaging /tmp/mylogger/project-code/target/scala-2.9.1/mylogger_2.9.1-1.0-SNAPSHOT.jar ... [info] Done packaging. [info] API documentation generation successful. [info] Packaging /tmp/mylogger/project-code/target/scala-2.9.1/mylogger_2.9.1-1.0-SNAPSHOT-javadoc.jar ... [info] Done packaging. [info] published mylogger_2.9.1 to /home/steve/development/play/play-2.0/framework/../repository/local/mylogger/mylogger_2.9.1/1.0-SNAPSHOT/poms/mylogger_2.9.1.pom [info] published mylogger_2.9.1 to /home/steve/development/play/play-2.0/framework/../repository/local/mylogger/mylogger_2.9.1/1.0-SNAPSHOT/jars/mylogger_2.9.1.jar [info] published mylogger_2.9.1 to /home/steve/development/play/play-2.0/framework/../repository/local/mylogger/mylogger_2.9.1/1.0-SNAPSHOT/srcs/mylogger_2.9.1-sources.jar [info] published mylogger_2.9.1 to /home/steve/development/play/play-2.0/framework/../repository/local/mylogger/mylogger_2.9.1/1.0-SNAPSHOT/docs/mylogger_2.9.1-javadoc.jar [info] published ivy to /home/steve/development/play/play-2.0/framework/../repository/local/mylogger/mylogger_2.9.1/1.0-SNAPSHOT/ivys/ivy.xml [success] Total time: 4 s, completed Mar 19, 2012 8:57:28 PMIf you take a look in $PLAY_HOME/repository/local, you’ll now see a directory called mylogger. Since we didn’t give an organisation name, the organisation is taken to be the same name as the module itself. Go into mylogger, and you’ll see the artifact – mylogger_2.9.1. The 2.9.1 part of the filename comes from Play itself, and is (appears to be) a versioning thing. If anyone knows more about this, please comment and let me know. Inside mylogger_2.9.1, we have the module version, which in this case is 1.0-SNAPSHOT, and this in turn contains jar files, source jars, Maven and Iyy info, etc. Where does all this information come from? It’s based on the project/Build.scala file. In here, you can give the name of the module, the organisation, the version, and various other pieces of information. For now, we’ll keep things as they are, but this extremely important file will be updated as we get deeper into some issues. 4. Providing a sample You can write the best, most incredibly useful module in the world, but without a sample application to a) show it works, and b) show HOW it works, you’re going to have trouble convincing people of that. This is why we changed the directory structure back when we first created the module. Open another terminal, and go to the mylogger/samples diretory – it’s time to show what mylogger can do. 4.1 A sample application is…a Play application Since we’re writing a Play module, it makes sense to provide a Play application as the sample. Now we’re in mylogger/samples, using “play new mylogger-sample” to create the sample application. Again, choose option 2 to make a simple Java application. 4.2 Declaring dependencies In order to use mylogger, we must declare a dependency for it in mylogger-sample/project/Build.scala. Open this file, and change val appDependencies = Seq( // Add your project dependencies here, )to val appDependencies = Seq( "mylogger" % "mylogger_2.9.1" % "1.0-SNAPSHOT" )You can see this matches the repository path of mylogger/mylogger_2.9.1/1.0-SNAPSHOT. In a brazen act of laziness, we’re also going to declare the local repository as the place to find our module. Change val main = PlayProject(appName, appVersion, appDependencies, mainLang = JAVA).settings( // Add your own project settings here ) to val main = PlayProject(appName, appVersion, appDependencies, mainLang = JAVA).settings( resolvers += "Local Play Repository" at "file://path/to/play-2.0/repository/local" ) (but change the path to be correct for your local installation) In the mylogger/samples/mylogger-sample directory, start up the Play console using “play”. If you use the “dependencies” command, you’ll see that mylogger is now a dependency of the application. IMPORTANT NOTE! Since we started the Play console after changing Build.scala, the changes were automatically picked up. If you change this file while the console is open, use “reload” to make sure the changes are used. 4.2 Using your module In your new default Play application, we’re going to add a single line to controllers/Application.java to call MyLogger: package controllers;import play.mvc.Controller; import play.mvc.Result; import views.html.index;public class Application extends Controller { public static Result index() { MyLogger.log("Here's my log message"); return ok(index.render("Your new application is ready.")); } }Note that we don’t need to import MyLogger as it’s also in the controllers package. “run” the application and go to http://localhost:9000. A brief time later, the page will render and in the console you’ll see [info] play - Application started (Dev) MyLogger: Here's my log messageReload the page a few times, and you’ll see the log message appear every time. 5. Have another beer Congratulations, you now have a module and a working sample. It doesn’t actually add much value to your sample app, but that’s something that will be addressed in part 2. In the meantime, head over the fridge and grab yourself another beer. You can download the complete source code for this example here Reference: Writing modules for Play 2, part 1: Get something working from our JCG partner Steve Chaloner at the Objectify blog....
selenium-logo

Integration Testing with Selenium

Overview I’ve been using this for sometime and I’ve come across a few things that appear to make life easier. I thought I’d share this as a tutorial, so I’ll walk you through these parts:Setting up a web project using Maven, configuring Selenium to run as an integration test on a C.I. Look into good ways to model the pages in your site using “page objects” and other ways to create points of protected-variation. Use JPA and Hibernate to perform CRUD operations on a database, and have Maven perform integration tests on them, without any of the costly and often undocumented set-up that this sometimes entails. This post assumes you’re comfortable with Java, Spring, Maven 2 and, of course, HTML. You’ll also want Firefox installed on you computer. This tutorial is intended to be otherwise technology agnostic. Creating a Webapp Firstly we’ll need a webapp to test. Create an project using the maven-webapp-archetype and call it “selenuim-tutorial”. To run integration tests (ITs) we’re going to use the Cargo plugin. This starts and stops containers such as Jetty and Tomcat. You can use Cargo to start your site using Jetty (its default) in one command without any changes: mvn cargo:runAnd check it in you browser at: http://localhost:8080/selenuim-tutorial You’ll get a 404 without welcome file set-up, so add that to the web.xml file: <welcome-file-list> <welcome-file>/index.jsp</welcome-file> </welcome-file-list> If you run cargo:run again you’ll now see the “Hello World!” page that was created by Maven. Configuring Cargo We can set-up Cargo to start a Jetty container prior to running the tests, and then stop it afterwards. This will allow us to start our site, run the integration tests, and then stop it afterwards. <plugin> <groupId>org.codehaus.cargo</groupId> <artifactId>cargo-maven2-plugin</artifactId> <version>1.2.0</version> <executions> <execution> <id>start</id> <phase>pre-integration-test</phase> <goals> <goal>start</goal> </goals> </execution> <execution> <id>stop</id> <phase>post-integration-test</phase> <goals> <goal>stop</goal> </goals> </execution> </executions> </plugin> You can test this work with: mvn verify One thing to note at this point is that Cargo runs on port 8080. If you’ve already got a process listening on that port you might see an error similar to this: java.net.BindException: Address already in use This might be because you’re already running another container on this port. If you want to run this on a C.I. (which may itself run on port 8080), this is likely to be something you’ll want to change. Add these lines to the plugin set-up: <configuration> <type>standalone</type> <configuration> <properties> <cargo.servlet.port>10001</cargo.servlet.port> </properties> </configuration> </configuration> Now the app will be here: http://localhost:10001/selenuim-tutorial/ Setting-up Integration Test Phase Next, we need to be able to run the integration tests. This requires the Maven failsafe plugin with appropriate goals added to your pom: <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>2.12</version> <executions> <execution> <id>default</id> <goals> <goal>integration-test</goal> <goal>verify</goal> </goals> </execution> </executions> </plugin> By default Failsafe expects tests to match the pattern “src/test/java/*/*IT.java”. Let’s create a test to demonstrate this. Note that I haven’t changed from Junit 3.8.1 yet. I’ll explain to why later on. Here’s a basic, incomplete test: package tutorial; import junit.framework.TestCase; public class IndexPageIT extends TestCase { @Override protected void setUp() throws Exception { super.setUp(); } @Override protected void tearDown() throws Exception { super.tearDown(); } public void testWeSeeHelloWorld() { fail(); } } Test that works: mvn verify You should see a single test failure. To test using Selenium you’ll need to add a test-scoped dependency to pom.xml: <dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-firefox-driver</artifactId> <version>2.19.0</version> <scope>test</scope> </dependency> We can now make a couple of changes to our test: import org.openqa.selenium.WebDriver; import org.openqa.selenium.firefox.FirefoxDriver; … private URI siteBase; private WebDriver drv; @Override protected void setUp() throws Exception { super.setUp(); siteBase = new URI("http://localhost:10001/selenuim-tutorial/"); drv = new FirefoxDriver(); } ... public void testWeSeeHelloWorld() { drv.get(siteBase.toString()); assertTrue(drv.getPageSource().contains("Hello World")); } We’ll remove these hard coded values later on. Run it again: mvn verify You shouldn’t see any failures. What you will have is a lingering Firefox. It won’t have closed. Run this test 100 times and you’ll have 100 Firefoxs running. This will quickly become a problem. We can resolve this by adding this initialisation block to our test: { Runtime.getRuntime().addShutdownHook(new Thread() { @Override public void run() { drv.close(); } }); } Naturally, if we create another test, we’ll soon be violating DRY principles. We’ll come to that in the next part, as well as looking at what happens when we require a database connection, and some other ways to make sure that your tests are simple to write and easy to maintain. Spring Context In the previous example, the URI for the app, and the driver used were both hard coded. Assuming you’re familiar with Spring context, this is a pretty straight forward to change these. Firstly we’ll add the correct dependencies: <dependency> <groupId>org.springframework</groupId> <artifactId>spring-context</artifactId> <version>3.1.1.RELEASE</version> <scope>test</scope> </dependency> This will allow us to use and application context to inject dependencies. But we’ll also need the correct Junit runner to test this, which can be found in the spring-test package: <dependency> <groupId>org.springframework</groupId> <artifactId>spring-test</artifactId> <version>3.1.1.RELEASE</version> <scope>test</scope> </dependency> We can now update our test to use this. Firstly we’ll need to create src/test/resources/applicationContext-test.xml <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd"> <bean id="siteBase" class="java.net.URI"> <constructor-arg value="http://localhost:10001/selenuim-tutorial/" /> </bean> <bean id="drv" class="org.openqa.selenium.firefox.FirefoxDriver" destroy-method="quit"/> </beans> Spring will clear up the browser when it finishes, so we can remove the shutdown hook from AbstractIT. This is more robust than having the test case do this. The spring-test doesn’t work with JUnit 3, it needs at least JUnit 4.5. Lets update to version 4.10 in our pom.xml: <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.10</version> <scope>test</scope> </dependency> Finally, we need to update our test to work with both Spring and JUnit 4.x: package tutorial; import static org.junit.Assert.assertTrue; import java.net.URI; import org.junit.Test; import org.junit.runner.RunWith; import org.openqa.selenium.WebDriver; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(locations = { "/applicationContext-test.xml" }) public class IndexPageIT { @Autowired private URI siteBase; @Autowired private WebDriver drv; @Test public void testWeSeeHelloWorld() { ... These changes moved the configuration from hard coded values into XML config. We can now change the location we are testing, e.g. to a different host, and change the web driver we’re using, which is left as an exercise for the user. A quick note on browsers. I’ve found that after a browser update, tests often start failing. There appears to be two solutions to this:Upgrade to the latest version of the web driver. Don’t upgrade the browser. I suspect the first option is the best in most cases, for security reasons Abstract IT Currently, you’ll need to duplicate all the code for IoC. A simple refactoring can sort this out. We’ll create a super-class for all tests, and pull-up common features. This refactoring uses inheritance rather than composition, for reasons I’ll cover later. package tutorial; import java.net.URI; import org.junit.runner.RunWith; import org.openqa.selenium.WebDriver; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(locations = { "/applicationContext-test.xml" }) public abstract class AbstractIT { @Autowired private URI siteBase; @Autowired private WebDriver drv; public URI getSiteBase() { return siteBase; } public WebDriver getDrv() { return drv; } }package tutorial; import static org.junit.Assert.assertTrue; import org.junit.Test; public class IndexPageIT extends AbstractIT { @Test public void testWeSeeHelloWorld() { getDrv().get(getSiteBase().toString()); assertTrue(getDrv().getPageSource().contains("Hello World")); } } Page Objects A “page object” is an object that encapsulates a single instance of a page, and provides a programatic API to that instance. A basic page might be: package tutorial; import java.net.URI; import org.openqa.selenium.WebDriver; public class IndexPage { /** * @param drv * A web driver. * @param siteBase * The root URI of a the expected site. * @return Whether or not the driver is at the index page of the site. */ public static boolean isAtIndexPage(WebDriver drv, URI siteBase) { return drv.getCurrentUrl().equals(siteBase); } private final WebDriver drv; private final URI siteBase; public IndexPage(WebDriver drv, URI siteBase) { if (!isAtIndexPage(drv, siteBase)) { throw new IllegalStateException(); } this.drv = drv; this.siteBase = siteBase; } } Note that I’ve provided a static method to return whether or we are at the index page, and I’ve commented it (debatably unnecessarily for such a self-documating method); page objects form an API and can be worthwhile documenting. You’ll also see that we throw an exception if the URL is incorrect. It’s worth considering what condition you use to identify pages. Anything that might change (e.g. the page title, which could change between languages) is probably a poor choice. Something unchanging and machine readable (e.g. the page’s path) are good choices; if you want to change the path, then you’ll need to change test. Now lets create ourself a problem. I’d like to add this to index.jsp, but the HTML produced is un-parsable: <% throw new RuntimeException(); %>Instead we’ll create a new servlet, but first we’ll need to add the servlet-api to the pom.xml: <dependency> <groupId>javax.servlet</groupId> <artifactId>servlet-api</artifactId> <version>2.5</version> <scope>provided</scope> </dependency>package tutorial; import java.io.IOException; import javax.servlet.ServletException; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; public class IndexServlet extends HttpServlet { private static final long serialVersionUID = 1L; protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { throw new RuntimeException(); } } Add it to the web.xml and remove the now unnecessary welcome page: <!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd" > <web-app> <servlet> <servlet-name>IndexServlet</servlet-name> <servlet-class>tutorial.IndexServlet</servlet-class> </servlet> <servlet-mapping> <servlet-name>IndexServlet</servlet-name> <url-pattern>/</url-pattern> </servlet-mapping> </web-app> Update IndexPageIT: @Test public void testWeSeeHelloWorld() { getDrv().get(getSiteBase().toString()); new IndexPage(getDrv(), getSiteBase()); } Run the test again. It passes. This might not be the behaviour you want. Selenium does not provide a way to check the HTTP status code via a WebDriver instance. Nor is the default error page sufficiently consistent between containers (compare this to what happens if you run on Tomcat for example); we cannot make assumptions about the error page’s content to figure out if an error occurred. Our index page currently does not have any machine readable features that allow us to tell it from an error page. To tidy up, modify the servlet to display index.jsp: protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { getServletContext().getRequestDispatcher("/index.jsp").forward(request, response); } Currently index.jsp is a little too simple. Create a new page named create-order.jsp alongside index.jsp, and create a link on index.jsp to that page. We can create a new class for the order page, and a method that navigates us from the index page to the order page. Add the following to index.jsp: <a href="create-order.jsp">Create an order</a> create-order.jsp can be blank for now. We can also create a page object for it: package tutorial; import java.net.URI; import org.openqa.selenium.WebDriver; public class CreateOrderPage { public static boolean isAtCreateOrderPage(WebDriver drv, URI siteBase) { return drv.getCurrentUrl().equals(siteBase.toString() + "create-order.jsp"); } private final WebDriver drv; private final URI siteBase; public CreateOrderPage(WebDriver drv, URI siteBase) { if (!isAtCreateOrderPage(drv, siteBase)) { throw new IllegalStateException(); } this.drv = drv; this.siteBase = siteBase; } } Add the following dependency to pom.xml which will give us some useful annotations: <dependency> <groupId>org.seleniumhq.selenium</groupId> <artifactId>selenium-support</artifactId> <version>2.19.0</version> <scope>test</scope> </dependency> We can flesh out IndexPage now: @FindBy(css = "a[href='create-order.jsp']") private WebElement createOrderLink; public IndexPage(WebDriver drv, URI siteBase) { if (!isAtIndexPage(drv, siteBase)) { throw new IllegalStateException(); } PageFactory.initElements(drv, this); this.drv = drv; this.siteBase = siteBase; } This call to PageFactory.initElements will populate fields annotated with @FindBy with the object matching the element on the web page. Note the use of a CSS selector, it’s to target the link in way that is unlikely to change. Other methods include matching elements on the page using the link text (which might change for different languages). We can now create a method on IndexPages which navigates to CreateOrderPages. public CreateOrderPage createOrder() { createOrderLink.click(); return new CreateOrderPage(drv, siteBase); } Finally we can create a test for this link in IndexPageIT: @Test public void testCreateOrder() { getDrv().get(getSiteBase().toString()); new IndexPage(getDrv(), getSiteBase()).createOrder(); assertTrue(CreateOrderPage.isAtCreateOrderPage(getDrv(), getSiteBase())); } Execute mvn verify and you should find the new test passes. At this point we have two tests that do not clean up between them. They use the same WebDriver instance for both tests, the last page will still be open and any cookies that were set will remain so. There are pros and cons of creating a single instance of a WebDriver for several tests. The main pro being reducing time cost of opening and closing browsers, but a con being that the browser is effectively left dirty after each test, cookies set, pop-ups open. We can make sure it is clean before each test with a suitable setUp method in AbstractIT: @Before public void setUp() { getDrv().manage().deleteAllCookies(); getDrv().get(siteBase.toString()); }There are alternative approaches to this, I’ll leave it up to you to look into ways of creating a new WebDriver instance prior the each test. The @FindBy annotation is especially useful when used on forms. Add a new form to create-order.jsp: <form method="post" name="create-order"> Item: <input name="item"/> <br/> Amount: <input name="amount"/><br/> <input type="submit"/> </form> Add those WebElements to CreateOrderPage , and a method to submit the form: @FindBy(css = "form[name='create-order'] input[name='item']") private WebElement itemInput; @FindBy(css = "form[name='create-order'] input[name='amount']") private WebElement amountInput; @FindBy(css = "form[name='create-order'] input[type='submit']") private WebElement submit; public CreateOrderPage(WebDriver drv, URI siteBase) { if (!isAtCreateOrderPage(drv, siteBase)) { throw new IllegalStateException(); } PageFactory.initElements(drv, this); this.drv = drv; this.siteBase = siteBase; } public CreateOrderPage submit(String item, String amount) { itemInput.sendKeys(item); amountInput.sendKeys(amount); submit.click(); return new CreateOrderPage(drv, siteBase); }Finally we can create a test for this: package tutorial; import static org.junit.Assert.*; import org.junit.Test; public class CreateOrderPageIT extends AbstractIT { @Test public void testSubmit() { new IndexPage(getDrv(), getSiteBase()).createOrder().submit("foo", "1.0"); } } Conclusion One thing you might note is that the submit method doesn’t require the amount to be a number as you might expect. You could create a test to see that submitting a string instead of a number. Integration tests can be time consuming to write and vulnerable to breaking as a result of changes to things such as the ID of an element, or name of an input. As a result the greatest benefit to be gained from creating them is initially create them just on business critical paths within your site, for example, product ordering, customer registration processes and payments. In the next part of this tutorial, we’ll looking at backing the tests with some data, and the challenges this engenders. Reference: Tutorial: Integration Testing with Selenium – Part 1 ,Tutorial: Integration Testing with Selenium – Part 2 from our JCG partner Alex Collins at the Alex Collins ‘s blog....
java-logo

IBM JVM tuning – gencon GC policy

This article will provide you detail on an important Java Heap space tuning consideration when migrating from a Java VM such as HotSpot or JRockit to the IBM JVM. This tuning recommendation is based on a recent troubleshooting and tuning mandate I performed for one of my IT clients. IBM JVM overview As you probably saw from my other articles, the IBM JVM is different than the HotSpot JVM in some aspects as it does not have a PermGen memory space for example. From a garbage collection perspective, it does provide you with advanced algorithms that can take advantage of a multi physical cores machine; similar to the HotSpot JVM. From a troubleshooting perspective, IBM provides you with many tools; including out-of-the-box Thread Dump and Heap Dump generation capabilities from its JVM implementation. The IBM JVM Thread Dump for example is particularly powerful as it provides you extra data on your JVM such as the active JVM environment variables, GC policies, loaded classes in each active class-loader etc. We will explore this in more detail on the part 4 of our Thread Dump Training plan. IBM VM – default GC behaviour Now back to our primary topic, it is very important that you understand the default behaviour of the IBM JVM garbage collector (version 1.5 & 1.6). By default, the Java Heap space is created using tenured memory only e.g. it does not create a separate YoungGen (nursery) space. This means that any memory allocation goes to the tenured space (short lived and long lived objects) which later gets collected by the default collector (via a Full GC). Find below a verbose GC snapshot showing you the default GC memory breakdown with explanations: <af type="tenured" id="5" timestamp="Mar 01 13:40:30 2012" intervalms="0.000"><minimum requested_bytes="48" /><time exclusiveaccessms="0.106" meanexclusiveaccessms="0.106" threads="0" lastthreadtid="0x000000011A846B00" /><tenured freebytes="20131840" totalbytes="2013265920" percent="0" ><soa freebytes="0" totalbytes="1993134080" percent="0" /><loa freebytes="20131840" totalbytes="20131840" percent="100" /></tenured><gc type="global" id="8" totalid="2492" intervalms="2017588.587"><finalization objectsqueued="199" /><timesms mark="808.286" sweep="9.341" compact="0.000" total="818.292" /><tenured freebytes="1362331024" totalbytes="2013265920" percent="67" ><soa freebytes="1344212368" totalbytes="1995147264" percent="67" /><loa freebytes="18118656" totalbytes="18118656" percent="100" /></tenured></gc><tenured freebytes="1362330976" totalbytes="2013265920" percent="67" ><soa freebytes="1344212320" totalbytes="1995147264" percent="67" /><loa freebytes="18118656" totalbytes="18118656" percent="100" /></tenured><time totalms="818.750" /></af>Ok, default IBM JVM GC policy is different… what is the problem? The problem with this default JVM policy is that all Java objects are copied to the tenured space and collected via a global collection (Full GC). For many Java EE applications, the ratio of short lived vs. long lived objects is much higher. This means that your JVM will need to perform quite a lot of major collections to clean up the short lived objects; results: increased frequency of Full GC, increased JVM pause time, increased CPU utilization and performance degradation! This is exactly what we observed while performing load testing following a migration to JVM HotSpot 1.5 (using incremental & parallel GC) to IBM JVM 1.6 with default GC policy. Heavy GC process was identified as the root cause as per the above explanation. Solution please! The good news is that the IBM JVM introduced generational & concurrent GC collector since version 1.5. This GC policy is providing exactly what we want:It does split the Java Heap space between nursery and tenured spaces Nursery (YoungGen) space objects are collected separately via the scavenger GC collector Tenured space objects are collected via the global GC collector The GC collector is concurrent and taking advantage of any multi physical cores machine Results:Reduced major collection frequency (Full GC) Reduced Full GC elapsed time & pause time  Increase JVM throughput  Increase performance & capacity of your application You can enable it by adding this JVM paremeter below: -Xgcpolicy:genconFind below what you can expect in your verbose GC log after enabling this GC policy:<af type="nursery" id="15" timestamp="Mar 08 05:34:06 2012" intervalms="1289096.227"><minimum requested_bytes="40" /><time exclusiveaccessms="0.085" meanexclusiveaccessms="0.085" threads="0" lastthreadtid="0x0000000118113900" /><refs soft="18043" weak="204057" phantom="27" dynamicSoftReferenceThreshold="32" maxSoftReferenceThreshold="32" /><nursery freebytes="0" totalbytes="530716672" percent="0" /><tenured freebytes="1887422016" totalbytes="2013265920" percent="93" ><soa freebytes="1786758720" totalbytes="1912602624" percent="93" /><loa freebytes="100663296" totalbytes="100663296" percent="100" /></tenured><gc type="scavenger" id="15" totalid="15" intervalms="1289097.271"><flipped objectcount="1486449" bytes="129908000" /><tenured objectcount="1176" bytes="184144" /><finalization objectsqueued="3082" /><scavenger tiltratio="73" /><nursery freebytes="364304408" totalbytes="495208448" percent="73" tenureage="10" /><tenured freebytes="1886766656" totalbytes="2013265920" percent="93" ><soa freebytes="1786103360" totalbytes="1912602624" percent="93" /><loa freebytes="100663296" totalbytes="100663296" percent="100" /></tenured><time totalms="233.886" /></gc><nursery freebytes="364238872" totalbytes="495208448" percent="73" /><tenured freebytes="1886766656" totalbytes="2013265920" percent="93" ><soa freebytes="1786103360" totalbytes="1912602624" percent="93" /><loa freebytes="100663296" totalbytes="100663296" percent="100" /></tenured><refs soft="17992" weak="5344" phantom="27" dynamicSoftReferenceThreshold="32" maxSoftReferenceThreshold="32" /><time totalms="236.858" /></af>Please keep in mind that it is still possible that your application may not benefit from this GC policy (bigger footprint of long lived objects etc.) so my recommendation to you is to always do your due diligence and perform proper capacity planning & load testing of your application before implementing any major tuning recommendations. Conclusion I hope this article has helped you understand the default IBM JVM 1.5/1.6 GC policy and how your Java EE application can benefit from this GC policy gencon tuning recommendation. Reference: IBM JVM tuning – gencon GC policy from our JCG partner Pierre-Hugues Charbonneau at the Java EE Support Patterns & Java Tutorial blog. ...
java-logo

Java EE Revisits Design Patterns: Observer

Aside from being implemented in many languages and many applications, Observer Pattern has been a part of Java since version 1.0. Observer Pattern is also a good implementation of Hollywood Principle. Just like Agents in Hollywood like to callback the candidates for a role instead of being called daily to be asked about available jobs, most of the server side resources like the push the available data to the appropriate client instead of being asked for updates on a time interval. Such time interval queries can be both consuming to much resource on the server and also cause more network traffic than actually needed. Although Java had support for Observer Pattern since day 0, it is always argued to be not the best implementation (Have a look at Observer and Observable). Being on JavaEE world may even complicate things. However JavaEE6 comes with an alternative. JavaEE6 offers ‘@Observable’ annotation as an easy out of box implementation of Observer Pattern. Lets visit the previous post and extend it to use observers. package com.devchronicles.observer;import javax.ejb.Stateless; import javax.ejb.TransactionAttribute; import javax.ejb.TransactionAttributeType; import javax.enterprise.event.Event; import javax.inject.Inject;/** * * @author Murat Yener */ @Stateless @TransactionAttribute(TransactionAttributeType.REQUIRED) public class EventService { @Inject private String message; @Inject Event<String> event; public void startService(){ System.out.println("start service call "+message); event.fire("this is my "+message); System.out.println("done..."); } }The EventService class will be injected an Event object of type String which can be used to fire String objects. If you had not read the previous post, message object is a String which will be produced by a factory and injected to the EventService class. To make it simpler you can just type any string constant to the variable called message. Since we are already done with the observable part, now it is time to create an observer to listen our events. package com.devchronicles.observer;import javax.ejb.Stateless; import javax.enterprise.event.Observes;/** * * @author Murat Yener */ @Stateless public class EventObserver {public void doLogging(@Observes String message){ System.out.println("Observed:"+message); } }The Observes annotation marks the method as an observer for fired String events. If you run the server and fire up start service method, you will realize how magically a string will be injected to EventService class and than fired where it will be caughed (observed) by EventObserver class. Surprisingly that is all you need to implement the observer pattern in JavaEE6. As seen before, JavaEE6 offers an easy way to implement the Observer Pattern. After publishing the post, I had receive few questions on how to differentiate string types that are fired and observed. Although in real world scenarios you wouldn’t probably firing and observing plain strings but your own objects which would be observed by their type, still it is pretty easy to differentiate same type of objects and setup different observers to listen them. First lets start with the part to differentiate plain strings. package com.devchronicles.observer;import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target; import javax.inject.Qualifier;/** * * @author Murat Yener */ @Qualifier @Retention(RetentionPolicy.RUNTIME) @Target({ElementType.FIELD,ElementType.PARAMETER}) public @interface MyEvent { Type value(); enum Type{ LOGGING, MESSAGE } }The interface above will act as annotation to mark the string to be fired and later to be observed just by annotating the appropriate parts. package com.devchronicles.observer;import javax.ejb.Stateless; import javax.ejb.TransactionAttribute; import javax.ejb.TransactionAttributeType; import javax.enterprise.event.Event; import javax.inject.Inject;/** * * @author Murat Yener */ @Stateless @TransactionAttribute(TransactionAttributeType.REQUIRED) public class EventService { @Inject private String message; @Inject @MyEvent(MyEvent.Type.LOGGING) Event<String> event;@Inject @MyEvent(MyEvent.Type.MESSAGE) Event<String> anotherEvent; public void startService(){ System.out.println("start service call "+message); event.fire("this is my "+message); System.out.println("done..."); anotherEvent.fire("done with the service!"); } }We just add MyEvent annotation with the desired type and later fire the events as we did before. The parts marked with red is all we added to the example in the previous post. Now lets annotate the observer part. Again We will be just adding the red parts to the previous example. package com.devchronicles.observer;import javax.ejb.Stateless; import javax.enterprise.event.Observes;/** * * @author Murat Yener */ @Stateless public class EventObserver {public void doLogging(@Observes @MyEvent(MyEvent.Type.LOGGING) String message){ System.out.println("Observed:"+message); }public void doLogging(@Observes @MyEvent(MyEvent.Type.MESSAGE) String message){ System.out.println("Observed another type of message:"+message); } }That would be all you would need to differentiate even the same type of objects to observe. Reference: JavaEE Revisits Design Patterns: Observer , JavaEE Revisits Design Patterns: Observer Part 2 from our JCG partner Murat Yener at the Developer Chronicles blog....
spring-logo

What is Spring Integration?

As the Spring Integration project slowly gains more adoption and interest, developers in the enterprise integration or enterprise development space will most likely come across it. They may find it interesting, but not fully understand what it is, what problems it was created to solve, where they can get more information, and where it fits within the open-source ecosystem of ESBs and other SOA infrastructure. Here’s my attempt at a normal-person’s description of what it is. Up first, what is it? It’s an open-source project commissioned by SpringSource to leverage the current capabilities of the Spring Framework to focus on the problems found in the application-integration space. Without a concrete example, or a more fundamental understanding, that last sentence is probably just as vague as the other information you may have seen about Spring Integration or integration in general. So let me go into a little more detail to help make that statement a little less vague. So why did the Spring folks decide to create a project specifically focused on integration? Doesn’t the Spring Framework itself provide a lot of that already? Spring does have fantastic abstractions for dealing with JMS, JDBC, transaction management, object-xml mapping, http/rmi invocation, and many others. It also provides a dependency-injection based framework which promotes cleaner, decoupled, and easier to test code. But if you take a step back and realize what Spring provides it really is just general purpose building blocks, a component model, that can be used in a limitless variety of solutions. So when it comes to system/application integration, you could implement your own, very capable, solution using these building blocks. However, application integration, and the problems inherent in solving those problems are not new. There are quite a few “patterns” that emerge once you’ve experienced a handful of attempts at integrating two systems for data exchange, process invocation, event notification, etc. These patterns were very well captured by Gregor Hohpe and Bobby Woolf in their timeless book “Enterprise Integration Patterns”. As I’m sure I’ve mentioned in previous blog posts, I highly recommend this book for anyone in the enterprise development space. These patterns are well known and proven for addressing most integration problems. The Spring folks decided to take the building blocks of their Spring Framework and the patterns presented in Hohpe’s book to create a more-focused framework that specifically deals with integrating applications. So what are the problems in the integration space? Like I said, they are described far better than I can in EIP, but here’s a simple description of a problem that is almost always present. Two applications need to share a piece of data, for example, a customer report which is originated on system A needs to be available in another system B. System A can only communicate with outside applications through a direct TCP connection and system B has a simple web-service for loading report information to it and is unwilling to change to anything else. How do you go about doing this? You could write some custom integration “glue code” that runs periodically: set up a batch process or a cron job, break out Java’s socket, or socket-nio libraries, connect up to system A, read and write to an input stream, grab the useful data, convert it to some kind of intermediary format, map some of that data to a SOAP xml message so that system B can understand it, break out AXIS or HttpCommons and send the xml to system B. A lot of the coding involved in creating this integration can be classified as infrastructure and not really “custom.” For example, connecting to TCP and reading/writing streams. Why should we have to write that code? There’s nothing custom about that. Delegate that to a framework/library. What about the polling to see whether an application is available? Delegate that too, that’s not a custom concern. And the webservice call? Generic components for TCP communication, polling or event handling, Web Service calls, routing and transforming, and many others. is exactly what Spring Integration provides. And it tries to mimic the full power of the patterns described in the EIP book while using a component model familiar to previous Spring Framework users. One question that I find comes up a lot in discussions about “what is Spring Integration” is how does it relate to an ESB or SOA architecture, and if one were to do an analysis about Spring Integration vs its competition, what specifically is its competition? First, Spring Integration is NOT an ESB. It is a *routing* and *mediation* framework. When i say it’s a mediation framework, I mean that it allows two different systems with different messages and protocols to talk with each other by “mediating” the message: resolve/negotiate differences between the two so they can exchange data. This mediation and routing framework can be used anywhere and doesn’t need to be deployed into a heavyweight ESB container, or any ESB container. It can be deployed within in application (either stand alone application or part of a Java EE solution within an application server), within an ESB if you need, as part of a message broker, etc. It’s flexible regarding deployment. Spring Integration itself should not be compared to ServiceMix, MuleESB, TIBCO, IBM or Oracle’s ESB solutions or other ESBs. One open-source project that comes to mind that would be a fair comparison is Apache’s Camel project which too is a mediation and routing engine. Apache Camel is also a very powerful and highly capable solution to the integration problem space and it also implements the patterns from the EIP book. I can do a comparison in a future blog post if readers show interest. For more information about Spring Integration, I recommend visiting their project page , taking a look at a recent book Pro Spring Integration, and of course reading and fully understanding the EIP book. Reference: What is Spring Integration? from our JCG partner Christian Posta at the Christian Posta Software blog....
java-logo

Threading stories: about robust thread pools

Another blog of my threading series. This time it’s about thread pools, robust thread pool settings in particular. In Java thread pools are implemented by the ThreadPoolExecutor class introduced in Java 5. The Javadoc of that class is very well organized. So I spare me the effort to give a general introduction here. Basically, what ThreadPoolExecutor does is, it creates and manages threads that process Runnable tasks that were submitted to a work queue by an arbitrary client. It’s a mechanism to perform work asynchronously, which is an important capability in times of multi-core machines and cloud computing. To be useful in a wide range of contexts ThreadPoolExecutor provides some adjustable parameters. This is nice, but it also leaves the decision to us, the developers, to choose the right settings for our concrete cases. Here is the largest constructor for a ThreadPoolExecutor. public ThreadPoolExecutor(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit, BlockingQueue<Runnable> workQueue, ThreadFactory threadFactory, RejectedExecutionHandler handler) { ... }Thread pool types Some of the parameters shown in the construtor above are very sensible ones in terms of resource consumption and resulting system stability. Based on the different parameter settings of the constructor it’s possible to distinguish some fundamental categories of thread pools. Here are some default thread pool settings offered by the Executors class. public static ExecutorService newCachedThreadPool() { return new ThreadPoolExecutor(0, Integer.MAX_VALUE, 60L, TimeUnit.SECONDS, new SynchronousQueue<Runnable>()); }public static ExecutorService newFixedThreadPool(int nThreads) { return new ThreadPoolExecutor(nThreads, nThreads, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>()); }In the “cached thread pool” the number of threads is unbounded. This is caused by a maximumPoolSize of Integer.MAX_VALUE in conjunction with SynchronousQueue. If you submit tasks in a burst to that thread pool, it will likely create a thread for each single task. In this scenario created threads terminate when they were idle for 60 seconds. The second example shows a “fixed thread pool”, where maximumPoolSize is set to a specific fixed value. The pools thread count will never exceed this value. If tasks arive in a burst and all threads are busy then they will queue up in the work queue (here a LinkedBlockingQueue). Threads in this fixed thread pool never die. The drawback of unbounded pools is obvious: both settings can cause JVM memory trouble (you get OutOfMemoryErrors – if you’re lucky). Let’s look at some bounded thread pool settings: ThreadPoolExecutor pool = new ThreadPoolExecutor(0, 50, 60, TimeUnit.SECONDS, new SynchronousQueue<Runnable>()); pool.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());ThreadPoolExecutor pool = new ThreadPoolExecutor(50, 50, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>(100000)); pool.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());The first snippet creates a chached thread pool with the number of threads bounded to a value of 50. If tasks arrive in a burst and all the threads are busy, a call to the ThreadPoolExecutor.execute() method would now be rejected by issueing a RejectedExecutionException. Often this isn’t what I typically want, therefore I change the saturation policy by setting the rejected-execution-handler to CallerRunsPolicy. This policy pushes the work back to the caller. That is, the client thread that issued the task for asynchronous execution will now run the task synchronously. You can develop your own saturation policy by implementing your own RejectedExecutionHandler. The second snippet creates a fixed thread pool with 50 threads and a work queue that is bounded to a value of 100000 tasks. If the work queue is full, the saturation policy pushes the work back to the client. The cached pool creates threads on demand and it terminates the threads if they idle for 60 seconds. The fixed pool keeps the threads alive. Thread pool boundaries As shown above there are two basic approaches to define thread pools: bounded and unbounded thread pools. Unbounded thread pools, like the default ones of Executors class work fine, as long you don’t submit too many tasks in a burst. If that happens unbounded thread pools can harm your systems stability. Either too many threads are created by a cached thread pool, or too many tasks are queued in a fixed thread pool. The letter is more difficult to achieve, but still possible. For production use it may be better to set the boundaries to some meaningfull values like in the last two thread pool settings. Because it can be tricky to define those “meaningfull boundaries” I have developed a little program that does that work for me. /** * A class that calculates the optimal thread pool boundaries. It takes the desired target utilization and the desired * work queue memory consumption as input and retuns thread count and work queue capacity. * * @author Niklas Schlimm * */ public abstract class PoolSizeCalculator {/** * The sample queue size to calculate the size of a single {@link Runnable} element. */ private final int SAMPLE_QUEUE_SIZE = 1000;/** * Accuracy of test run. It must finish within 20ms of the testTime otherwise we retry the test. This could be * configurable. */ private final int EPSYLON = 20;/** * Control variable for the CPU time investigation. */ private volatile boolean expired;/** * Time (millis) of the test run in the CPU time calculation. */ private final long testtime = 3000;/** * Calculates the boundaries of a thread pool for a given {@link Runnable}. * * @param targetUtilization * the desired utilization of the CPUs (0 <= targetUtilization <= 1) * @param targetQueueSizeBytes * the desired maximum work queue size of the thread pool (bytes) */ protected void calculateBoundaries(BigDecimal targetUtilization, BigDecimal targetQueueSizeBytes) { calculateOptimalCapacity(targetQueueSizeBytes); Runnable task = creatTask(); start(task); start(task); // warm up phase long cputime = getCurrentThreadCPUTime(); start(task); // test intervall cputime = getCurrentThreadCPUTime() - cputime; long waittime = (testtime * 1000000) - cputime; calculateOptimalThreadCount(cputime, waittime, targetUtilization); }private void calculateOptimalCapacity(BigDecimal targetQueueSizeBytes) { long mem = calculateMemoryUsage(); BigDecimal queueCapacity = targetQueueSizeBytes.divide(new BigDecimal(mem), RoundingMode.HALF_UP); System.out.println("Target queue memory usage (bytes): " + targetQueueSizeBytes); System.out.println("createTask() produced " + creatTask().getClass().getName() + " which took " + mem + " bytes in a queue"); System.out.println("Formula: " + targetQueueSizeBytes + " / " + mem); System.out.println("* Recommended queue capacity (bytes): " + queueCapacity); }/** * Brian Goetz' optimal thread count formula, see 'Java Concurrency in Practice' (chapter 8.2) * * @param cpu * cpu time consumed by considered task * @param wait * wait time of considered task * @param targetUtilization * target utilization of the system */ private void calculateOptimalThreadCount(long cpu, long wait, BigDecimal targetUtilization) { BigDecimal waitTime = new BigDecimal(wait); BigDecimal computeTime = new BigDecimal(cpu); BigDecimal numberOfCPU = new BigDecimal(Runtime.getRuntime().availableProcessors()); BigDecimal optimalthreadcount = numberOfCPU.multiply(targetUtilization).multiply( new BigDecimal(1).add(waitTime.divide(computeTime, RoundingMode.HALF_UP))); System.out.println("Number of CPU: " + numberOfCPU); System.out.println("Target utilization: " + targetUtilization); System.out.println("Elapsed time (nanos): " + (testtime * 1000000)); System.out.println("Compute time (nanos): " + cpu); System.out.println("Wait time (nanos): " + wait); System.out.println("Formula: " + numberOfCPU + " * " + targetUtilization + " * (1 + " + waitTime + " / " + computeTime + ")"); System.out.println("* Optimal thread count: " + optimalthreadcount); }/** * Runs the {@link Runnable} over a period defined in {@link #testtime}. Based on Heinz Kabbutz' ideas * (http://www.javaspecialists.eu/archive/Issue124.html). * * @param task * the runnable under investigation */ public void start(Runnable task) { long start = 0; int runs = 0; do { if (++runs > 5) { throw new IllegalStateException("Test not accurate"); } expired = false; start = System.currentTimeMillis(); Timer timer = new Timer(); timer.schedule(new TimerTask() { public void run() { expired = true; } }, testtime); while (!expired) { task.run(); } start = System.currentTimeMillis() - start; timer.cancel(); } while (Math.abs(start - testtime) > EPSYLON); collectGarbage(3); }private void collectGarbage(int times) { for (int i = 0; i < times; i++) { System.gc(); try { Thread.sleep(10); } catch (InterruptedException e) { Thread.currentThread().interrupt(); break; } } }/** * Calculates the memory usage of a single element in a work queue. Based on Heinz Kabbutz' ideas * (http://www.javaspecialists.eu/archive/Issue029.html). * * @return memory usage of a single {@link Runnable} element in the thread pools work queue */ public long calculateMemoryUsage() { BlockingQueue<Runnable> queue = createWorkQueue(); for (int i = 0; i < SAMPLE_QUEUE_SIZE; i++) { queue.add(creatTask()); } long mem0 = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory(); long mem1 = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory(); queue = null; collectGarbage(15); mem0 = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory(); queue = createWorkQueue(); for (int i = 0; i < SAMPLE_QUEUE_SIZE; i++) { queue.add(creatTask()); } collectGarbage(15); mem1 = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory(); return (mem1 - mem0) / SAMPLE_QUEUE_SIZE; }/** * Create your runnable task here. * * @return an instance of your runnable task under investigation */ protected abstract Runnable creatTask();/** * Return an instance of the queue used in the thread pool. * * @return queue instance */ protected abstract BlockingQueue<Runnable> createWorkQueue();/** * Calculate current cpu time. Various frameworks may be used here, depending on the operating system in use. (e.g. * http://www.hyperic.com/products/sigar). The more accurate the CPU time measurement, the more accurate the results * for thread count boundaries. * * @return current cpu time of current thread */ protected abstract long getCurrentThreadCPUTime();}The program will find ideal thread pool boundaries for the maximum capacity of your work queue and the required thread count. The algorithms are based on the work of Brian Goetz and Dr. Heinz Kabutz, you can find the references in the Javadoc. Calculating the capacity required by your work queue in a fixed thread pool is relatively simple. All you need is the desired target size of the work queue in bytes divided by the average size of your submitted tasks in bytes. Unfortunately, calculating the maximum thread count *isn’t* an exact science. However, if you use the formulas in the program you avoid the harmful extremes of too large work queues and too many threads. Calculating the ideal pool size depends on the wait time to compute time ratio of your task. The more wait time, the more threads required to achieve a given utilization. The PoolSizeCalculator requires the desired target utilization and the desired maximum work queue memory consumption as input. Based on an investigation of object sizes and CPU time it returns the ideal settings for maximum thread count and work queue capacity in the thread pool. Let’s go through an example. The following snippet shows how to use the PoolSizeCalculator in a scenario of 1.0 (=100%) desired utilization and 100000 bytes maximum work queue size. public class MyPoolSizeCalculator extends PoolSizeCalculator {public static void main(String[] args) throws InterruptedException, InstantiationException, IllegalAccessException, ClassNotFoundException { MyThreadSizeCalculator calculator = new MyThreadSizeCalculator(); calculator.calculateBoundaries(new BigDecimal(1.0), new BigDecimal(100000)); }protected long getCurrentThreadCPUTime() { return ManagementFactory.getThreadMXBean().getCurrentThreadCpuTime(); }protected Runnable creatTask() { return new AsynchronousTask(0, "IO", 1000000); } protected BlockingQueue<Runnable> createWorkQueue() { return new LinkedBlockingQueue<>(); }}MyPoolSizeCalculator extends the abstract PoolSizeCalculator. You need to implement three template methods: getCurrentThreadCPUTime, creatTask, createWorkQueue. The snippet applies standard Java Management Extensions for the CPU time measurement (line 13). If JMX isn’t accurate enough, then other frameworks can be considered (e.g. SIGAR API). Thread pools work best when tasks are homogeneous and independent. Therefore the createTask-method creates an instance of a single type of Runnable task (line 17). This task will be investigated to calculate wait time to CPU time ratio. Finally, I need to create an instance of the work queue to calculate memory usage of submitted task (line 21). The output of that program shows the ideal settings for the work queue capacity and the maximum pool size (number of threads). These are the results for my examplary I/O intense AsynchronousTask on a dual core machine. Target queue memory usage (bytes): 100000 createTask() produced com.schlimm.java7.nio.threadpools.AsynchronousTask which took 40 bytes in a queue Formula: 100000 / 40 * Recommended queue capacity (bytes): 2500 Number of CPU: 2 Target utilization: 1.0 Elapsed time (nanos): 3000000000 Compute time (nanos): 906250000 Wait time (nanos): 2093750000 Formula: 2 * 1.0 * (1 + 2093750000 / 906250000) * Optimal thread count: 6.0The ‘recommended queue capacity’ and the ‘optimal thread count’ are the important values. An ideal setting for my AsynchronousTask would be as follows: ThreadPoolExecutor pool = new ThreadPoolExecutor(6, 6, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>(2500)); pool.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());Using those settings your work queue cannot grow larger then the desired 100000 bytes. And since the desired utilization is 1.0 (100%), it does not make sense to make the pool larger then 6 threads (wait time to compute time ratio is three – for each compute time intervall, three wait time intervalls follow). The results of the program largely depend on the type of tasks you process. If the tasks are homogenous and compute intense the program will likely recommend to set the pool size to the number of available CPU. But if the task has wait time, like in I/O intense tasks, the program will recomend to increase the thread count to achieve 100% utilization. Also notice, some tasks change their wait time to compute time ratio after some time of processing, e.g. if the file size of an I/O operation increases. This fact suggests to develop a self-tuning thread pool (one of my follow up blogs). In any case, you should make the thread pool sizes configurable, so that you can adjust them at runtime. OK, that’s all in terms of robust thread pools for now. I hope you enjoyed it a little. And don’t blame me if the formula isn’t 100% accurate in terms of maximum pool size. As I said, it’s not an exact science, it’s about getting an idea of the ideal pool size. Reference: “Threading stories: about robust thread pools” from our JCG partner Niklas....
devops-logo

Monitoring : Making sense of the buzz word jungle

Nowadays we are constantly reminded of the virtues of being proactive or more colloquially put “Being one step ahead of the game” when it comes to handling our businesses whether it be a SME or a multi-national cooperation. Quickly detecting or in some cases even predicting, trends in activities originating within and outside the organization and streamlining business activities accordingly may decide between death or life, of the business it’s said. The often touted solution for this problem is implementing a proper monitoring solution which would give the decision makers relevant information at correct time. However most businesses are at a loss where to begin or how to properly implement means of obtaining such insights. This is not surprising given that even the buzzwords surrounding the monitoring concepts tend to be fuzzy. Whoa.. That’s some pretty serious language (OK it is, at least to me . I consider my self linguistically challenged when it comes to English.). Well I wanted to start with a serious note since we are dealing with a serious subject here right??. . Anyway this says a part of the story when it comes to business monitoring domain. Sometimes the monitoring solutions forced on businesses are just like this. Some serious mumbo jumbo with hundreds of bells and whistles which most of us don’t care to understand. And of course some times not capturing what really needs to be monitored in the business as well. On top of that there is a buzz word soup surrounding the monitoring products which each vendor come up with different interpretations according to their implementations. Anyway let’s get some perspective on some business monitoring key words according to the way I see it. Let’s monitor some activities “Business Activity Monitoring” is a term coined by Gartner Inc. which is defined as the “The aggregation, analysis and presentation of real-time information about activities inside organizations and involving customers and partners”. However it can be seen the term is used in different contexts meaning different things to different people specially when it comes vendor solutions. The confusion tends be mostly around the fact on what can be considered a business activity. For example for a business executive a sale of a product will be a perfectly valid business activity which need to be monitored while for tech op guy would need monitoring on the load of the server hosting the sales application. I have heard some people say the latter does not really falls under the term “Business Activity” since that level of monitoring is of no importance to strategic decision-making of the business. But as far as I believe it is no less important and should be a part of a comprehensive monitoring solution since any high level decisions made would depend on the smooth functioning of daily operations supported by a proper functioning infrastructure (If servers are out sales numbers are going to get hurt. So will the sales projections. Simple as that). It’s a matter of providing a suitable view to each intended user group according to the type of monitoring information they are interested in. Anyway latter kind of monitoring may better fit under “Operational Intelligence” category of which I will be talking about in a bit. In that sense we can think of “Business Activity Monitoring” as a subset of “Business Monitoring” so that this fulfills a part of the holistic view on approaching the monitoring problem where all of what needs to be monitored in the business would come under a comprehensive monitoring solution. This is one major point where the vendors differ in their solutions. Some monitoring solutions focus on a mixture of monitoring aspects and so their definition of BAM varies accordingly. BPM – A side kick?? Another difference between various BAM solutions is in the way they are implemented. Some assume the presence of an existence of a Business Process Management(BPM) solution, mostly from the same vendor and so the monitoring solution is tightly coupled to that. While these kinds of solutions may provide better integration in terms of the products in my opinion they lack the flexibility to monitor most business scenarios where no business process management solutions are in place. If the monitoring framework is generic enough it’s a matter of putting required data capturing agents at points of interest to capture and send data to the BAM solution which should be able to correlate events from incoming events. However if there is a BPM solution already present from the same vendor it should also be able to leverage that as well. This way it would provide most flexibility in terms of monitoring requirements. Key to success – KPI Another term mentioned side by side with BAM is key performance indicators(KPI). A BAM solution would monitor a set of predefined KPIs and make sure that necessary actions are taken (it may be firing some alerts to relevant parties or even automatically triggering some corrective action if possible) when KPIs are not met with respect to their desired values. A good definition that I found on what constitute a KPI is as follows. Key Performance Indicators are quantifiable measurements that reflect the critical success factors of an organization. They will differ depending on the organization So these are highly specific to the organization. Let me give a couple of simple examples on KPIs.For a retail store a valid KPI would be the percentage of days where daily sales revenue target was not met. For a delivery service a KPI would monitor the number of deliveries that went 10% overtime than their expected delivery times. A KPI for a call center would monitor the number of calls which took less than 2 minutes to resolve the problem.Here we can identify the importance of the ability to customize the KPI definitions according to the nature of the business. While properly identifying the necessary KPIs should be done with involvement of the business management, the BAM solution should facilitate defining business specific KPI definitions. Intelligence in Operations – OI Next comes the “Operational Intelligence” aspect of the business monitoring. It is more or less similar to “Business Activity Monitoring” except that “Operational Intelligence” is more oriented towards monitoring day today business activities and geared to find issues in the system in real-time in order for taking corrective actions. I believe technical operations monitoring fits under this description since it involves the day-to-day aspect and the required response times for any found issue should be more real-time. But business matrices requiring close monitoring may well be included as part of “Operational Intelligence” aspects as well. So here comes another word (“Real time”) in to the mix which means different things to different people. There are several levels of real-timeness as per products we see in the market. Some position them as real-time monitoring solutions while others support near real time monitoring and the boundary between these are blurry at best. As with any thing else when it comes to monitoring, the required response time of the solution depends on the context. A solution monitoring a business critical application server may require response times within several seconds while a low volume internal application server may not need such real-time monitoring. A good rule of thumb should be that if it’s real-time expect a sub minute response time while if it’s near real-time a couple of minutes lag at times may be acceptable. Of course the vendors can stretch these either way according to their implementations. So always try to read between the lines of marketing terms to really see whether the solution a vendor is proposing really matches what is required. CEP to the rescue Often the response times required by “Operational Intelligence” monitoring necessitates the usage of a Complex Event Processing(CEP) solution underneath which would monitor incoming event streams upon entry and trigger certain actions when anomalies are detected. So the real-timeness of the product will directly depend upon the performance characteristics and scalability of the CEP solution used underneath. Another type of Intelligence – BI Next type of “Intelligence” a business want is “Business Intelligence”. Yeah I know there are so many types of “Intelligences” floating around and this is one of the important ones. This is geared towards finding trends in business operations and market environment and coming up with predictions on the business conditions. This is basically a historical data analysis which may pull out data from a data ware house do some ETL operations and run some data mining operations on data to gain new insights on business operations. So these jobs are not real-time rather batch jobs which are scheduled at suitable intervals. Ok. I think that’s enough for a day. Hope I made some sense out of the monitoring buzz word fiesta. Hopefully this post would be good base for a next post I plan to write some time soon in which I would outline some practical experiences me and our team had while implementing a business monitoring solution ourselves. Reference: Monitoring : Making sense of the buzz word jungle from our JCG partner Buddhika Chamith at the Source Open blog....
json-logo

MOXy as Your JAX-RS JSON Provider – Server Side

In a previous series of posts I covered how EclipseLink JAXB (MOXy) can be leveraged to create a RESTful data access service. In this post I will cover how easy it is to leverage MOXy’s new JSON binding on the server side to add support for JSON messages based on JAXB mappings.MOXy as Your JAX-RS JSON Provider – Server Side MOXy as Your JAX-RS JSON Provider – Client SideWhy EclipseLink JAXB (MOXy)? Below are some of the advantages of using MOXy as your JSON binding provider:Widest support for JAXB annotations among JSON binding providers. Support for both XML and JSON: Binding to JSON & XML – Geocode Example. MOXy contains extensions such as @XmlInverseReference for mapping JPA entities to JSON and XML: Part 3 – Mapping JPA entities to XML (using JAXB). External mapping document as an alternative to annotations: MOXy’s XML Metadata in a JAX-RS Service. CustomerService The message types that a JAX-RS service understands is controlled using the @Produces and @Consumes annotations. In this post I have specified that all the operations now support “application/json” in addition to “application/xml”. A more detailed description of this service is available in the following post: Creating a RESTful Web Service – Part 4/5. package org.example; import java.util.List; import javax.ejb.*; import javax.persistence.*; import javax.ws.rs.*; import javax.ws.rs.core.MediaType; @Stateless @LocalBean @Path("/customers") public class CustomerService { @PersistenceContext(unitName="CustomerService", type=PersistenceContextType.TRANSACTION) EntityManager entityManager; @POST @Consumes({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) public void create(Customer customer) { entityManager.persist(customer); } @GET @Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) @Path("{id}") public Customer read(@PathParam("id") long id) { return entityManager.find(Customer.class, id); } @PUT @Consumes({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) public void update(Customer customer) { entityManager.merge(customer); } @DELETE @Path("{id}") public void delete(@PathParam("id") long id) { Customer customer = read(id); if(null != customer) { entityManager.remove(customer); } } @GET @Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON}) @Path("findCustomersByCity/{city}") public List<Customer> findCustomersByCity(@PathParam("city") String city) { Query query = entityManager.createNamedQuery("findCustomersByCity"); query.setParameter("city", city); return query.getResultList(); } }MOXyJSONProvider We will implement a JAX-RS MessageBodyReader/MessageBodyWriter to plugin support for MOXy’s JSON binding. This implementation is generic enough that it could be used as is to enable JSON-binding for any JAX-RS service using MOXy as the JAXB provider. Some interesting items to note:There are no compile time dependencies on MOXy.  The eclipselink.media-type property is used to enable JSON binding on the unmarshaller (line 34) and marshaller (line 55). The eclipselink.json.include-root property is used to indicate that the @XmlRootElement annotation should be ignored in the JSON binding (lines 35 and 56). When creating the JAXBContext the code first checks to see if a JAXBContext has been registered for this type (lines 70 and 71). This is useful if you want to leverage MOXy’s external mapping document: MOXy’s XML Metadata in a JAX-RS Service.package org.example; import java.io.*; import java.lang.annotation.Annotation; import java.lang.reflect.*; import javax.xml.transform.stream.StreamSource; import javax.ws.rs.*; import javax.ws.rs.core.*; import javax.ws.rs.ext.*; import javax.xml.bind.*; @Provider @Produces(MediaType.APPLICATION_JSON) @Consumes(MediaType.APPLICATION_JSON) public class MOXyJSONProvider implements MessageBodyReader<Object>, MessageBodyWriter<Object>{ @Context protected Providers providers; public boolean isReadable(Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) { return true; } public Object readFrom(Class<Object> type, Type genericType, Annotation[] annotations, MediaType mediaType, MultivaluedMap<String, String> httpHeaders, InputStream entityStream) throws IOException, WebApplicationException { try { Class<?> domainClass = getDomainClass(genericType); Unmarshaller u = getJAXBContext(domainClass, mediaType).createUnmarshaller(); u.setProperty("eclipselink.media-type", mediaType.toString()); u.setProperty("eclipselink.json.include-root", false); return u.unmarshal(new StreamSource(entityStream), domainClass).getValue(); } catch(JAXBException jaxbException) { throw new WebApplicationException(jaxbException); } } public boolean isWriteable(Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) { return true; } public void writeTo(Object object, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType, MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream) throws IOException, WebApplicationException { try { Class<?> domainClass = getDomainClass(genericType); Marshaller m = getJAXBContext(domainClass, mediaType).createMarshaller(); m.setProperty("eclipselink.media-type", mediaType.toString()); m.setProperty("eclipselink.json.include-root", false); m.marshal(object, entityStream); } catch(JAXBException jaxbException) { throw new WebApplicationException(jaxbException); } } public long getSize(Object t, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) { return -1; } private JAXBContext getJAXBContext(Class<?> type, MediaType mediaType) throws JAXBException { ContextResolver<JAXBContext> resolver = providers.getContextResolver(JAXBContext.class, mediaType); JAXBContext jaxbContext; if(null == resolver || null == (jaxbContext = resolver.getContext(type))) { return JAXBContext.newInstance(type); } else { return jaxbContext; } } private Class<?> getDomainClass(Type genericType) { if(genericType instanceof Class) { return (Class<?>) genericType; } else if(genericType instanceof ParameterizedType) { return (Class<?>) ((ParameterizedType) genericType).getActualTypeArguments()[0]; } else { return null; } } }Server Setup If you are using GlassFish as your application server then you need to replace the following EclipseLink bundles with their counterpart from an EclipseLink 2.4 install.org.eclipse.persistence.antlr.jar org.eclipse.persistence.asm.jar org.eclipse.persistence.core.jar org.eclipse.persistence.jpa.jar org.eclipse.persistence.jpa-modelgen.jar org.eclipse.persistence.moxy.jar org.eclipse.persistence.oracle.jarFurther Reading If you enjoyed this post then you may also be interested in:RESTful Services MOXy as Your JAX-RS JSON Provider – Client Side Creating a RESTful ServicePart 1 – The Database Part 2 – Mapping the Database to JPA Entities Part 3 – Mapping JPA entities to XML (using JAXB) Part 4 – The RESTful Service Part 5 – The ClientMOXy’s XML Metadata in a JAX-RS Service JSON BindingJSON Binding with EclipseLink MOXy – Twitter Example Binding to JSON & XML – Geocode ExampleApplication Server IntegrationGlassFish 3.1.2 is Full of MOXy (EclipseLink JAXB) EclipseLink MOXy is the JAXB Provider in WebLogic Server 12cReference: MOXy as Your JAX-RS JSON Provider – Server Side from our JCG partner Blaise Doughan at the Java XML & JSON Binding blog....
java-logo

JRockit jrcmd tutorial

This article will provide you an overview and tutorial on how you can perform an initial analysis and problem isolation of a JRockit Java Heap problem using the jrcmd tool. A deeper analysis and tutorial using JRockit Mission Control and Heap Dump analysis (JRockit version R28+ only) will be covered in future articles. For a quick overview of the JRockit Java Heap Space, please consult the article below: JRockit Java Heap Space JRCMD tool overview jrcmd is a free tool that is available out-of-the-box in the JRockit binaries. It allows you generate and collect crucial data from your runtime JRockit VM such as:Java process memory space breakdown (Java Heap vs Native memory spaces) Java Heap diagnostic (histogram) – Java loaded classes On-demand JRockit Heap Dump generation (version R28+ only) Thread Dump generation More… For this article, we created a simple Java program leaking internally. We will use this program to demonstrate how you can leverage jrcmd to perform your initial analysis. Sample Java memory leak program This simple Java program is simply adding String data to a static HashMap and slowly leaking to the point of the JVM running out of Java Heap memory. This program will allow you to visualize a slowly growing Java Heap leak via JRockit jrcmd. Please note that a Java Heap size of 128 MB (-Xms128m –Xmx128m) was used for this example. /** * JavaHeapLeakSimulator * @author Pierre-Hugues Charbonneau * http://javaeesupportpatterns.blogspot.com */ public class JavaHeapLeakSimulator { private final static int NB_ITERATIONS = 500000000; // ~1 KB data footprint private final static String LEAKING_DATA_PREFIX = "datadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadata datadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadata datadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadata datadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadata datadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadata datadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadatadata"; // Map used to stored our leaking String instances private static Map<String, String> leakingMap;static { leakingMap = new HashMap<String, String>(); } /** * @param args */ public static void main(String[] args) { System.out.println("Java Heap Leak Simulator 1.0"); System.out.println("Author: Pierre-Hugues Charbonneau"); System.out.println("http://javaeesupportpatterns.blogspot.com/"); try { for (int i = 0; i < NB_ITERATIONS; i++) {String data = LEAKING_DATA_PREFIX + i; // Add data to our leaking Map data structure... leakingMap.put(data, data);// Slowdown the Java program so we can monitor the leak before the OutOfMemoryError condition Thread.sleep(1); }} catch (Throwable any) {if (any instanceof java.lang.OutOfMemoryError) { System.out.println("OutOfMemoryError triggered! " + any.getMessage() + " [" + any + "]"); } else { System.out.println("Unexpected Exception! " + any.getMessage() + " [" + any + "]"); } }System.out.println("JavaHeapLeakSimulator done!"); } }JRCMD – initial execution JRCMD can be executed from the local server hosting the JVM that you want to monitor or remotely via JRockit Mission Control. The executable is located within the JRockit JDK that you are using: <JRockit_JDK_HOME>/bin/jrcmdThe default jrcmd execution will return the list of active JRockit Java process Id that you can monitor: C:\Apps\Weblogic1035\jrockit_160_24_D1.1.2-4\bin>jrcmd 5360 org.ph.javaee.tool.oom.JavaHeapLeakSimulator 5952 6852 jrockit.tools.jrcmd.JrCmdJRCMD – Java Heap monitoring The next step is to start monitoring the Java Heap memory usage and histogram. A Java Heap histogram is a snapshot of the biggest pools of Java Class instances. This allow you to pinpoint the leaking data type. Ple You can either chose between print_object_summary (quick summary) or heap_diagnostics (complete breakdown). C:\Apps\Weblogic1035\jrockit_160_24_D1.1.2-4\bin>jrcmd 5360 heap_diagnostics Invoked from diagnosticcommand ======== BEGIN OF HEAPDIAGNOSTIC ========================= Total memory in system: 8465022976 bytes Available physical memory in system: 5279170560 bytes -Xmx (maximal heap size) is 134217728 bytes Heapsize: 134217728 bytes Free heap-memory: 123592704 bytes--------- Detailed Heap Statistics: --------- 90.9% 3948k 5468 +3948k [C 3.0% 128k 5490 +128k java/lang/String 2.1% 92k 3941 +92k java/util/HashMap$Entry 1.2% 50k 461 +50k java/lang/Class 0.8% 35k 21 +35k [Ljava/util/HashMap$Entry; 0.6% 24k 7 +24k [B 0.3% 15k 305 +15k [Ljava/lang/Object; 0.3% 14k 260 +14k java/net/URL 0.2% 6k 213 +6k java/util/LinkedHashMap$Entry 0.1% 4k 211 +4k java/io/ExpiringCache$Entry 0.1% 2k 4 +2k [Ljrockit/vm/FCECache$FCE; 0.0% 1k 50 +1k [Ljava/lang/String; 0.0% 1k 10 +1k java/lang/Thread 0.0% 1k 61 +1k java/util/Hashtable$Entry 0.0% 1k 7 +1k [I 0.0% 0k 19 +0k java/util/HashMap 0.0% 0k 19 +0k java/lang/ref/WeakReference 0.0% 0k 7 +0k [Ljava/util/Hashtable$Entry; 0.0% 0k 19 +0k java/util/Locale 0.0% 0k 11 +0k java/lang/ref/SoftReference 0.0% 0k 1 +0k [S …………………………………………………- The first column correponds to the Class object type contribution to the Java Heap footprint in % – The second column correponds to the Class object type memory footprint in K – The third column correponds to the # of Class instances of a particular type – The fourth column correponds to the delta – / + memory footprint of a particular type As you can see from the above snapshot, the biggest data type is [C (char in our case) & java.lang.String. In order to see which data types are leaking, you will need to generate several snapshots. The frequency will depend of the leaking rate. In our example, find below another snapshot taken after 5 minutes: # After 5 minutes --------- Detailed Heap Statistics: --------- 93.9% 26169k 28746 +12032k [C 2.4% 674k 28768 +295k java/lang/String 2.3% 637k 27219 +295k java/util/HashMap$Entry 0.9% 259k 21 +128k [Ljava/util/HashMap$Entry; 0.2% 50k 462 +0k java/lang/Class 0.1% 24k 7 +0k [B# After 5 more minutes --------- Detailed Heap Statistics: --------- 94.5% 46978k 50534 +20809k [C 2.4% 1184k 50556 +510k java/lang/String 2.3% 1148k 49007 +510k java/util/HashMap$Entry 0.5% 259k 21 +0k [Ljava/util/HashMap$Entry; 0.1% 50k 462 +0k java/lang/ClassThe third & fourth column are showing a constant increase. As you can see, the leaking data in our case are [C, java.lang.String and java.util.HashMap$Entry which all increased from ~4 MB to 28 MB, 50 MB and growing… It is easy to pinpoint the leaking data type(s) with this approach but what about the source (root cause) of the leaking data type(s)? This is where jrcmd is no longer useful. Deeper memory leak analysis will require you to use either JRockit Mission Control or Heap Dump analysis (JRockit R28+ only). A final point, before you can conclude on a true Java Heap leak, please ensure that jrcmd snapshots are taken after at least one Full GC in between the captures (what you are interested in is OldGen leak e.g. Java objects surviving major GC collections). JRCMD Thread Dump generation Thread Dump analysis is crucial for stuck Thread related problems but can also be useful to troubleshoot certain types of Java Heap problem. For example, it can pinpoint the source of a sudden Java Heap increase by exposing the culprit Thread(s) allocating a large amount of memory on the Java Heap in a short amount of time. Thread Dump can be generated using jrcmd print_threads option. ** Thread Dump captured from our sample Java program after removing the Thread.sleep() and increasing the Java Heap capacity ** C:\Apps\Weblogic1035\jrockit_160_24_D1.1.2-4\bin>jrcmd 5808 print_threads 5808:===== FULL THREAD DUMP =============== Mon Apr 09 09:08:08 2012 Oracle JRockit(R) R28.1.3-11-141760-1.6.0_24-20110301-1429-windows-ia32 "Main Thread" id=1 idx=0x4 tid=6076 prio=5 alive, native_blocked at jrockit/vm/Allocator.getNewTla(II)V(Native Method) at jrockit/vm/Allocator.allocObjectOrArray(Allocator.java:354)[optimized] at java/util/Arrays.copyOfRange(Arrays.java:3209)[inlined] at java/lang/String.<init>(String.java:215)[inlined] at java/lang/StringBuilder.toString(StringBuilder.java:430)[optimized] at org/ph/javaee/tool/oom/JavaHeapLeakSimulator.main(JavaHeapLeakSimulator.java:38) at jrockit/vm/RNI.c2java(IIIII)V(Native Method) -- end of trace ……………………………………….We can see that our sample Java program is creating a lot of java.lang.String objects from the “Main Thread” executing our JavaHeapLeakSimulator program. Conclusion I hope this article has helped you understand you can leverage the JRockit jrcmd tool for quick Java Heap analysis. I’m looking forward for your comments and questions. Future articles will include a deeper JRockit Java Heap and Heap Dump analysis tutorial. Reference: JRockit jrcmd tutorial from our JCG partner Pierre-Hugues Charbonneau at the Java EE Support Patterns & Java Tutorial blog....
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below:
Close