Enterprise Java

How to do Continuous Integration with Java 8, NetBeans Platform 8, Jenkins, Jacoco and Sonar

Intro

Java 8 is there, the promised revolution is finally released, and I am sure that a lot of you are having in mind the same question “Should I use it in my project?”.

Well, I had the same question for few months and today that I have an answer I would like to share it with you. A lot of aspects have been influencing this decision but in this post I want to focus on one in particular that is:

Can I continue to do Continuous Integration with Java 8 and NetBeans Platform?

 
The main question was around the maturity of the tools necessary to do CI, and how easy was to integrate that with the ant build scripts of the NetBeans Platform.

Fortunately, we found that it is possible and easy to do!

I would also thanks Alberto Requena Sanchez for his contribution on this article.

The Technical Environment

Working in a project where Safety & Quality are the main drivers, CI is vital.

For this reason I started with my team a “proof of concept” to show that the following technologies were ready to work together:

  • Java 8, NetBeans 8.0 & Ant
  • JUnit 4 & Jacoco 0.7.1
  • Jenkins & Sonar 4.2

Scope of this post is to explain all the steps done to install and setup the necessary tools to have a completely working CI server for Java 8. Note that the proof has been done on a developer machine on Windows 7, but is easy to do the same in a Linux server.

The next diagram shows at high level the architecture that will be described in the post.

Continuous integration diagram
Continuous integration diagram

 

Java 8, NetBeans 8.0 & Ant

Java 8 is released, get it here, install it, study it (preferable) and start to use it!

We are using the NetBeans Platform 8.0 to create a modular application. This application has a Multi Layered Architecture where each layer is a Suite of Modules, and where the final executable is just an integrated set of Suites.

We are using Ant to build our projects, but if you are using Maven the procedure can even be simplified since the Sonar integration in Jenkins can be done via a plugin that uses Maven.

JUnit 4 & Jacoco 0.7.1

Naturally, we are doing unit tests, and for this reason we use JUnit 4. It is well integrated everywhere, specially in NetBeans.

Jacoco is a great tool for the generation of code coverage and since version 0.7.1 it fully supports Java 8.

Jenkins & Sonar 4.2

Jenkins is the engine of our CI server, it will integrate with all the above described technologies without any issue. The tested version is 1.554.

Sonar is doing all the quality analysis of the code. The release 4.2 has a full compatibility with Java 8.

Using Sonar with Ant needs a small library that contains the target to be integrated in Jenkins. If you are using Maven instead you can just install the plugin for Maven.

Starting the puzzle

Step 1 – NetBeans

  1. Install Java 8 & NetBeans 8.0
  2. Create a module suite with several modules, several classes and several jUnit tests
  3. Commit the code into your source code version management server
  4. Inside the harness of NetBeans
  5. Create a folder in the harness named “jacoco-0.7.1″ containing the downloaded jacoco jars
  6. Create a folder in the harness named “sonar-ant-task” and put inside the downloaded sonar ant jars
  7. Create a file in the harness named sonar-jacoco-module.xml and paste the following code inside:
  8. <?xml version="1.0" encoding="UTF-8"?>
    <!--
    
    -->
    <project name="sonar-jacoco-module" basedir="." xmlns:jacoco="antlib:org.jacoco.ant" xmlns:sonar="antlib:org.sonar.ant">
    <description>Builds the module suite otherSuite.</description>
    
    <property name="jacoco.dir" location="${nbplatform.default.harness.dir}/jacoco-0.7.1"/>
    <property name="result.exec.file" location="${jacoco.dir}/jacoco.exec"/>
    <property name="build.test.results.dir" location="build/test/unit/results"/>
    
    <property file="nbproject/project.properties"/>
    
    <!-- Step 1: Import JaCoCo Ant tasks -->
    <taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml">
    <classpath path="${jacoco.dir}/jacocoant.jar"/>
    </taskdef>
    
    <!-- Target at the level of modules -->
    <target name="-do-junit" depends="test-init">
    <echo message="Doing testing for jacoco" />
    <macrodef name="junit-impl">
    <attribute name="test.type"/>
    <attribute name="disable.apple.ui" default="false"/>
    <sequential>
    <jacoco:coverage destfile="${build.test.results.dir}/${code.name.base}_jacoco.exec">
    <junit showoutput="true" fork="true" failureproperty="tests.failed" errorproperty="tests.failed" 
    filtertrace="${test.filter.trace}" tempdir="${build.test.@{test.type}.results.dir}" timeout="${test.timeout}">
    <batchtest todir="${build.test.@{test.type}.results.dir}">
    <fileset dir="${build.test.@{test.type}.classes.dir}" includes="${test.includes}" excludes="${test.excludes}"/>
    </batchtest>
    <classpath refid="test.@{test.type}.run.cp"/>
    <syspropertyset refid="test.@{test.type}.properties"/>
    <jvmarg value="${test.bootclasspath.prepend.args}"/>
    <jvmarg line="${test.run.args}"/>
    <!--needed to have tests NOT to steal focus when running, works in latest apple jdk update only.-->
    <sysproperty key="apple.awt.UIElement" value="@{disable.apple.ui}"/>
    <formatter type="brief" usefile="false"/>
    <formatter type="xml"/>
    </junit>
    </jacoco:coverage>
    <copy file="${build.test.results.dir}/${code.name.base}_jacoco.exec" todir="${suite.dir}/build/coverage"/>
    <!--
    Copy the result of all the unit tests of all the modules into one common
    folder at the level of the suite, so that sonar could find those files to
    generate associated reports
    -->
    <copy todir="${suite.dir}/build/test-results">
    <fileset dir="${build.test.results.dir}">
    <include name="**/TEST*.xml"/>
    </fileset>
    </copy>
    <fail if="tests.failed" unless="continue.after.failing.tests">Some tests failed; see details above.</fail>
    </sequential>
    </macrodef>
    <junit-impl test.type="${run.test.type}" disable.apple.ui="${disable.apple.ui}"/>
    </target>
    
    </project>

    Scope of this file is to override the do-junit task adding the jacoco coverage, and to copy the result of the unit test of each module in the build of the suite, so that sonar will find all of them together to perform its analysis.

  9. Create a file in the harness named sonar-jacoco-suite.xml and paste the following code inside
  10. <?xml version="1.0" encoding="UTF-8"?>
    <project name="sonar-jacoco-suite" basedir="." xmlns:jacoco="antlib:org.jacoco.ant" xmlns:sonar="antlib:org.sonar.ant">
    <description>Builds the module suite otherSuite.</description>
    
    <property name="jacoco.dir" location="${nbplatform.default.harness.dir}/jacoco-0.7.1"/>
    <property name="result.exec.file" location="build/coverage"/>    
    
    <!-- Define the SonarQube global properties (the most usual way is to pass these properties via the command line) -->
    <property name="sonar.jdbc.url" value="jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8" />
    <property name="sonar.jdbc.username" value="sonar" />
    <property name="sonar.jdbc.password" value="sonar" />
    <!-- Define the SonarQube project properties -->
    <property name="sonar.projectKey" value="org.codehaus.sonar:example-java-ant" />
    <property name="sonar.projectName" value="Simple Java Project analyzed with the SonarQube Ant Task" />
    <property name="sonar.projectVersion" value="1.0" />
    <property name="sonar.language" value="java" />
    <!-- Load the project properties file for retrieving the modules of the suite -->
    <property file="nbproject/project.properties"/>
    
    <!-- Using Javascript functions to build the paths of the data source for sonar configuration -->
    <script language="javascript"> 
    <![CDATA[
    
    // getting the value
    modulesName = project.getProperty("modules");
    modulesName = modulesName.replace(":",",");
    res = modulesName.split(",");
    srcModules = "";
    binariesModules = "";
    testModules = "";
    //Build the paths  
    for (var i=0; i<res.length; i++)
    {
    srcModules += res[i]+"/src,";
    binariesModules += res[i]+"/build/classes,";
    testModules += res[i]+"/test,";
    }
    //Remove the last comma
    srcModules = srcModules.substring(0, srcModules.length - 1);
    binariesModules = binariesModules.substring(0, binariesModules.length - 1);
    testModules = testModules.substring(0, testModules.length - 1);
    // store the result in a new properties
    project.setProperty("srcModulesPath",srcModules);
    project.setProperty("binariesModulesPath",binariesModules);
    project.setProperty("testModulesPath",testModules);
    ]]>
    </script>  
    <!-- Display the values -->       
    <property name="sonar.sources" value="${srcModulesPath}"/>
    <property name="sonar.binaries" value="${binariesModulesPath}" />
    <property name="sonar.tests" value="${testModulesPath}" />
    <!-- Define where the coverage reports are located -->
    <!-- Tells SonarQube to reuse existing reports for unit tests execution and coverage reports -->
    <property name="sonar.dynamicAnalysis" value="reuseReports" />
    <!-- Tells SonarQube where the unit tests execution reports are -->
    <property name="sonar.junit.reportsPath" value="build/test-results" />
    <!-- Tells SonarQube that the code coverage tool by unit tests is JaCoCo -->
    <property name="sonar.java.coveragePlugin" value="jacoco" />
    <!-- Tells SonarQube where the unit tests code coverage report is -->
    <property name="sonar.jacoco.reportPath" value="${result.exec.file}/merged.exec" />
    <!--  Step 1: Import JaCoCo Ant tasks  -->
    <taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml">
    <classpath path="${jacoco.dir}/jacocoant.jar"/>
    </taskdef>    
    <target name="merge-coverage">        
    <jacoco:merge destfile="${result.exec.file}/merged.exec">
    <fileset dir="${result.exec.file}" includes="*.exec"/>
    </jacoco:merge>
    </target>
    
    <target name="sonar">
    <taskdef uri="antlib:org.sonar.ant" resource="org/sonar/ant/antlib.xml">
    <!-- Update the following line, or put the "sonar-ant-task-*.jar" file in your "$HOME/.ant/lib" folder -->
    <classpath path="${harness.dir}/sonar-ant-task-2.1/sonar-ant-task-2.1.jar" />
    </taskdef>
    
    <!-- Execute the SonarQube analysis -->
    <sonar:sonar />
    </target>
    
    </project>

    Scope of this file is to define at the level of the suite the sonar configuration and the sonar ant task. If you are using for sonar some special database or special users is here that you must change the configuration.

    Another task that is defined is the jacoco merge that will actually take all the generated exec for each module and merge them into one single exec in the build of the suite, to permit sonar to make its analysis.

  11. Replace the content of the build.xml of each module with this one:
  12. <description>Builds, tests, and runs the project com.infrabel.jacoco.</description>
    <property file="nbproject/suite.properties"/>
    <property file="${suite.dir}/nbproject/private/platform-private.properties"/>
    <property file="${user.properties.file}"/>
    <import file="${nbplatform.default.harness.dir}/sonar-jacoco-module.xml"/>
    <import file="nbproject/build-impl.xml"/>
  13. Replace the content of the build.xml of each suite with this one:
  14. <description>Builds the module suite otherSuite.</description>
    <property file="nbproject/private/platform-private.properties"/>
    <property file="${user.properties.file}"/>
    <import file="${nbplatform.default.harness.dir}/sonar-jacoco-suite.xml"/>
    <import file="nbproject/build-impl.xml"/>

    Step 2 – Jenkins

  15. In “Manage Jenkins -> Manage Plugins” go in the available list and install (if not already present) the following plugins:
    • JaCoCo
    • Mercurial or Subversion
    • Sonar

    If you are behind a firewall or proxy and getting issue to configure the network settings you can always download and install them manually from here. In this case remember to download also the dependencies of each plugin first.

  16. In “Manage Jenkins -> Configure System” check that all plugins are correctly setup, see the following screenshots to have an example (replace the folders with the good ones for you):
  17. jenkins-jdk1

    jenkins-jdk

    jenkins-sonar-1

    jenkins-sonar-2

  18. Create a new free style project, configure the version control of your preference and in the “Build” panel add the following three “Invoce Ant” tasks:
  19. jenkins-configure

  20. Finally in the “Post-build Actions” panel add a new “Record Jacoco Coverage Report” configured like this one:
  21. jenkins-jacoco

    Step 3 – Sonar

  22. Create a database following this script, and optionally run this query to make the connection work:
  23. GRANT ALL PRIVILEGES ON 'sonar'.* TO 'sonar'@'localhost';
  24. Go in the configuration file of sonar (sonar.properties) and enable the use of MySQL, the file is located in the conf folder of the installation
  25. # Permissions to create tables, indices and triggers 
    # must be granted to JDBC user.
    # The schema must be created first.
    sonar.jdbc.username=sonar
    sonar.jdbc.password=sonar
    
    #----- MySQL 5.x
    # Comment the embedded database and uncomment the following 
    # line to use MySQL
    sonar.jdbc.url=jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true
  26. In the configuration of sonar update the java plugin if necessary to be compatible with Java 8
  27. If necessary go and configure your proxy always in the sonar.properties file

Done!

Now everything is set-up, you can go in NetBeans, do a build, commit your code, then in Jenkins launch the build and after the build is ok check the project in Sonar.

That’s all! I hope I did not forget anything, but in case if you find some errors during the process do not hesitate to leave a comment, I will try to find the solution.

Marco Di Stefano

Marco is a software engineer specialized in software architecture and process automation. He has been involved in all the software v-cycle phases and actually he works for the railway industry.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

4 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Oliver Milke
Oliver Milke
9 years ago

Nice Guide!

I especially like the combination of Jenkins and Sonar, which is the same setup I introduced to my team’s infrastructure – except for the fact we already used maven before introducing CI.

I just wonder which are your reasons to manage code coverage within Jenkins. Sonar also provides reporting on Code Coverage, thus having the build completely separated from all inspections and, in addition, all inspections in one place.

So, what are the benefits you see in using the JaCoCo-Plugin of Jenkins over using Sonar for code coverage analysis?

Marco Di Stefano
9 years ago
Reply to  Oliver Milke

Hi Oliver, thanks for your comment. it is true that Sonar already provides jacoco features, but in our team we find it very interesting to have this small resume also in Jenkins. Jenkins is used as a first check by developers to see if introduced failures in the build or in the test. Having it on jacoco also insures that no regression has been introduced by a succes build. Not all developers consult sonar daily, but jenkins yes. Also for reporting and making diagrams we have always been using jenkins. But as you say, is just a choice, since you… Read more »

Oliver Milke
Oliver Milke
9 years ago

This is an interesting insight, thanks for sharing your thoughts.

I am considering to also introduce some reports into Jenkins now :)

Might be that this is also a good idea since the people over at sonarsource seem to plan a change on Code Coverage with 4.3 anyway.

rogerio
rogerio
9 years ago

Quote:

“””
9. Replace the content of the build.xml of each module with this one:

10. Replace the content of the build.xml of each suite with this one:
“”””

Question: Where I encounter the file build.xml? In my project (ant task build.xml) or harness folder?

Back to top button