What's New Here?


A Camel Demo for Amazon’s Simple Worklfow Service

In a previous post I explained why AWS SWF service is good and announced the new Camel SWF component. Now the component documentation is ready and here is a simplistic fully working demo. It consist of three independent standalone Camel routes: A workflow producer allows us to interact with a workflow. It can start a new workflow execution, query its state, send signals to a running workflow, or terminate and cancel it. In our demo, the WorkflowProducer starts a route that schedules 10 workflow executions where the each execution receives as an argument a number.       package com.ofbizian.swf.demo;import org.apache.camel.builder.RouteBuilder; import org.apache.camel.main.Main;public class WorkflowProducer {public static String COMMON_OPTIONS = "accessKey=XXX" + "&secretKey=XXX" + "&domainName=demo" + "&workflowList=demo-wlist" + "&activityList=demo-alist" + "&version=1.0" + "&clientConfiguration.endpoint=swf.eu-west-12.amazonaws.com";public static void main(String[] args) throws Exception { Main main = new Main(); main.enableHangupSupport(); WorkflowProducerRoute route = new WorkflowProducerRoute(); main.addRouteBuilder(route); main.run(); }static class WorkflowProducerRoute extends RouteBuilder {@Override public void configure() throws Exception {from("timer://workflowProducer?repeatCount=10") .setBody(property("CamelTimerCounter")) .to("aws-swf://workflow?" + COMMON_OPTIONS + "&eventName=calculator") .log("SENT WORKFLOW TASK ${body}"); } } } Once a workflow execution is scheduled, we need a process that will decide what are the next steps for it. In Camel it is done using a Workflow Consumer. A workflow consumer represents the workflow logic. When it is started, it will start polling workflow decision tasks and process them. In addition to processing decision tasks, a workflow consumer route, will also receive signals (send from a workflow producer) or state queries. The primary purpose of a workflow consumer is to schedule activity tasks for execution using activity producers. Actually activity tasks can be scheduled only from a thread started by a workflow consumer. package com.ofbizian.swf.demo;import org.apache.camel.builder.RouteBuilder; import org.apache.camel.component.aws.swf.SWFConstants; import org.apache.camel.main.Main;public class WorkflowConsumer {public static void main(String[] args) throws Exception { Main main = new Main(); main.enableHangupSupport(); WorkflowConsumerRoute route = new WorkflowConsumerRoute(); main.addRouteBuilder(route); main.run(); }static class WorkflowConsumerRoute extends RouteBuilder {@Override public void configure() throws Exception {from("aws-swf://workflow?" + WorkflowProducer.COMMON_OPTIONS + "&eventName=calculator").choice() .when(header(SWFConstants.ACTION).isEqualTo(SWFConstants.SIGNAL_RECEIVED_ACTION)) .log("Signal received ${body}").when(header(SWFConstants.ACTION).isEqualTo(SWFConstants.GET_STATE_ACTION)) .log("State asked ${body}").when(header(SWFConstants.ACTION).isEqualTo(SWFConstants.EXECUTE_ACTION)).setBody(simple("${body[0]}")) .log("EXECUTION TASK RECEIVED ${body}").filter(simple("${body} > 5")) .to("aws-swf://activity?" + WorkflowProducer.COMMON_OPTIONS + "&eventName=incrementor"); } }} The logic in our demo decider is simple: if the incoming argument is greater than 5, we schedule a task for execution. Otherwise the workflow will complete as there are no other tasks to be executed. Notice that it also has branches for handing signal and state query events. The final peace of our distributed (since it consists of three independent applications) workflow application is the ActivityConsumer that actually performs some calculations. It has the simplest possible implementation: increments the given argument and returns it. package com.ofbizian.swf.demo;import org.apache.camel.builder.RouteBuilder; import org.apache.camel.main.Main;public class ActivityConsumer {public static void main(String[] args) throws Exception { Main main = new Main(); main.enableHangupSupport(); ActivityConsumerRoute route = new ActivityConsumerRoute(); main.addRouteBuilder(route); main.run(); }static class ActivityConsumerRoute extends RouteBuilder {@Override public void configure() throws Exception { from("aws-swf://activity?" + WorkflowProducer.COMMON_OPTIONS + "&eventName=incrementor") .setBody(simple("${body[0]}")) .log("RECEIVED ACTIVITY TASK ${body}") .setBody(simple("${body}++")); } } } All you need to do to run this demo is to create the appropriate workflow domain and add your key/secret to the route.   Reference: A Camel Demo for Amazon’s Simple Worklfow Service from our JCG partner Bilgin Ibryam at the OFBIZian blog. ...

Kanban from a Trench

Introduction After forming part of a software project team that got underway with a significant project at the BBC Worldwide in 2013, I felt that it would be useful to release a few notes about my experience of Kanban and how it helps to deliver good quality software to customers. Here are some notes that I prepared in order to deliver a Kanban motivational speech to an engineering community on a subsequent project outside of the BBC. Kanban from a Trench Kanban is technique for managing the software development process. It doesn’t tell us how to develop software but provides techniques that help us to enforce what we, (as a software development team), agree is the right formula to develop acceptable quality software for our customers. As an example, we may determine that an adequate software engineering process requires the following:Well understood and agreed requirements to be the foundation of onward development. Development is sympathetic to an evolving well understood architecture that’s published for the entire team to work from. We practice JIT development/engineering – only do what is required when it’s required. A set of coherent, well planned and engineered tests are built and maintained – from unit tests to UAT. Deployment and release management is actively managed through planning and coordination. Feature development priority adjustments are possible in order that our agile team can respond quickly to change instigated by key stake holders. Peer review of all team assets – code, documentation, user manuals and tests – as a minimum. Feature designs are documented with light-weight descriptions and pictures – bullet points and photos of whiteboards. Sometimes, formal UML diagrams can be justified, i.e. for state or sequence representations. The development process must evolve with the team, product and organisational aspirations.The Kanban Board For the Kanban projects in which I have been involved, our project board typically consists of five key areas:a) a team pool (a collection of names), b) a group of associated columns that represent our software engineering process, c) some streams, d) a set of feature tasks in various stages and e) a calendar of significant events.a) Team Pool: The team pool is a collection of names/avatars, each one a picture and name of a person whom is available for work on tasks – each team member usually has two avatars. You can work on at least one task at a time, often we have capacity to take on another task in parallel. These avatars help teach us about how we work as a team. For example, If an attempt is made to assign an avatar to more than two tasks, or is very frequently switched between tasks before tasks are finished – the board is very likely telling us something that we need to understand and deal with. Maybe we have insufficient resources in the team, maybe we have knowledge concentrated in one person (exclusive knowledge). Regardless, we need to record this and if it continues to happen, take some corrective action to resolve it. b) Group of Associated Columns: These columns represent phases or major milestones in our software development process. Kanban does not prescribe these columns as listed here below but suggests what might be a good starting point for our team to move forward from – the process will grow and evolve with the team.As an example, the following columns were used for the BBC Worldwide project in which I was involved: Inventory – A collection of features that will have a broad level description in order that it can be understood (three-point estimation only at this stage). No further detail is required right now – remember, JIT. Identify – Features that we’ve agreed to spend some further time on. We need to understand if the feature is date driven, what the value proposition is, how to understand if the goal is achieved or not (once it’s deployed we need to measure success), identify significant dependencies (other work/functions etc). Which release version/date is targeted etc. Analyse and Design – During this stage we will define and refine acceptance criteria – these will be written on the card. We now need to gather sufficient requirements such that the customer will get what they want (and hopefully asked for) and we understand how to build it. We will discuss and record BDD scenarios in preparation of QA. Analysis and design will be peer reviewed with a minimum agreed number of team members whom have business, technical and architectural knowledge relevant to the feature undergoing development. If necessary, the feature will be broken down into smaller units/tasks, each consisting of work that will last (approximately) between 0.5 and 5 days. Wireframes (from the whiteboard) will be drawn, described and agreed/approved by the customer. The feature will be reviewed in terms of architecture. The general test approach will be agreed, also consider technical debt, i.e. missing tests for existing code packages. Agree documentation of the feature, it’s setup, configuration, user manual entries etc. Agree to commit to build the feature. Develop and QA – Each new class, service, technical feature must have been written, have passing tests and executable documentation where required. Automated tests will be created and triggered by schedule. Tasks will not leave this column until they have been added into ‘develop’ (a Git branch) and all tests green-ticked. Ready to Deploy – Team agrees that they’re happy for the feature to go into the next release and a demo to the customer is conducted. One of the key aspects of Kanban is Work In Progress (WIP) limits. Kanban tells us that it’s better not to take on too much and that we should always consider finishing things in preference to starting new ones. This is often quite a challenge for teams new to Kanban because it means instead of taking on a new task, you may have to go and pair with someone that’s already working on a task. We all know that pairing should be encouraged, this is one route towards that. Pairing work may be coding, testing, documentation, QA, infrastructure, etc. WIP limits are defined for each column, they may have different values depending on the team size, columns and our experience of what works. We can increase WIP limits but beware consequences such as blocking. Instead of increasing WIP limits, get work finished instead. It’s often convenient to decompose a column into two parts, one to show in-progress tasks and one to show those that have been completed but are not ready to move into the next column. c) Some streams: It’s sometimes easier to track and read the board if groups of related tasks are bunched together in a stream. If during analyse and design a feature is broken down into, e.g. 8 sub-tasks, each of these sub tasks can be grouped into a single stream so we can readily see they are related.d) Feature tasks: The focus of the Kanban board is the tasks that represent feature development. The aim is to pull tasks through the system (from the right-hand side) rather than push them in from the left. Each task card contains a progress history. It’s not unusual for cards to have extensions attached to the reverse of them, design diagrams, JIRA numbers, dependencies, dates for movement along the board. To further aid recognition, it’s quite useful to use colour coded cards such that feature work, documentation, infrastructure and bugs can be easily recognised. It’s really important that the Kanban work board is maintained up to date, so this is something that usually takes place each morning – a stand-up. Whomever is running the stand-up will go through the tasks on the board and ask for an update from any person working on the task. e) Project Calendar: The entire team needs to know if somebody is sick, on holiday or that a release is imminent. How do we do maintenance? We could consider that two classes of bug exist, those directly related to features currently being undertaken and those related to historical feature development. Bugs related to current feature development should be fixed by the feature crew/team working on it, up to and including the User Acceptance Test phase, i.e. post deployment. The primary difference between product feature development and historical bug fixing seems to be stability of day-to-day work tasks and their duration – frequently changing priorities (intra-day) and generally, very short develop/test life-cycle. Ideally, we’d expect features to undergo development, in part or whole, during a period of days or weeks. Context switching is expensive so we want engineers to remain focused on the feature(s) that they’re working on rather than frequently switching between features and historic bugs. Therefore, having two project boards (or two streams on a single board), one representing core feature development and one representing bugs could suit quite well. The aspects that are different in these two regards, that need separating, are the units of work, rather than team members. Therefore, it’s a sensible approach to enable team members to be available to work on both core feature development and historical bugs as work-loads and priorities dictate. Existing Technical Debt We need to work out a mechanism that promotes, encourages and tracks necessary refactoring in order that it’s well socialised, publicised and coordinated. We could track this as a stream on the main board or use a distinct Kanban board to track it? Estimation For Inventory tasks, three-point estimation will be used. Best case, worst case and expected. We’ll use this to help drive which features to take on first given an arbitrary set of features that require development for a given release. This estimation can be further refined when breaking a task down in the Analysis and Design stages. This information will help feed into a release tracking board that represents feature dependencies and release dates. Architecture At any stage of any feature or release, it should be possible to view an architecture diagram to use for alignment of feature development. Interesting aspects are current architecture, aspirational next step architecture and final state architecture. Of course all three of these will move over time as the product and its environment matures. As with all other work, an architectural conjecture will be proposed and presented to the team as part of the general review process. None of this work is hidden, it will all appear as tasks on the Kanban board. Demos It should have been made clear from column exit criteria, but running a demo for the customer or his/her representative is a pre-requisite to a feature being added to a release. GitFlow At what stage does software make it into ‘develop’, release and support. The Kanban board needs to be well aligned with our release process, or vice-versa. Following a series of probing and explorative questions, the team commenced trying to understand what it would take in order to establish engineering satisfaction for software production. What does the team consider as adequate product engineering? Collectively, the team have identified the following criteria as those that are essential to building a product of sufficient quality such that we are happy to give it to our customers. The groupings are somewhat artificial, but give us an idea of the type and spread of areas for which we have collectively, accumulated important engineering criteria.The Kanban Board The physical Kanban board will be the master copy of feature development status. We need to build and place a board in an area that’s always accessible, as several team members have observed it would not be a good idea to have this located in a meeting room. JIRA tickets will be created for the highest-level feature names but at the end of the process, i.e. when they hit the last column (I.e. UAT). Generally, we will not get bogged down in keeping JIRA up-to-date with the physical board – communication is key and that can’t be replaced by an issue tracking system so let’s not try. From the previous examples, we will now define the columns that we’re going to use and assign exit criteria to each of them. Board Management Although, as a software development team collectively, we will own the process, the Kanban board needs to be actively managed along with the teams using the board. Sometimes we will require arbitration, encouragement, steering and decisions in order to operate. If an agile coach is not available, we need to identify someone to adopt the role of coach or manager. Acknowledgements I’d like to say a big thanks to Sabina Kamber Salamanca and Kevin Ryan for all of the knowledge sharing and guidance that led me to discover a better way to engineer software – inspirational.   Reference: Kanban from a Trench from our JCG partner Matt Vickery at the GreenDot Software blog. ...

Using Google GSON : Extra Goodies : Part I

Introduction This is follow-up article to previous Getting Started with Google GSON where it shows the introductory material on using the Google Gson. This article shows some extra goodies of GSON library. As there are lot of things to write about these extra goodies, so i splitted long article into 2 article series, So, this part one of it and in coming next article, i will be posting rest of remaining stuff. The beauty in using GSON library is that, it does not require any annotations or configurations for simple conversions. Although it makes use of default configurations to keep things simple. GSON has flexibility in creating the Json from Java object and vice-versa. Our truly Gson class contains lot of defaults for json conversion but we can override the defaults with use of class GsonBuilder. So, lets get started. Pretty Formatted JSON output When you print the json output string created using Gson then you will see output in one line, something like following {"name":"ajduke","languagesKnown":["Java","Scala","JavaScript"]} This is default formatting mode called Compact mode. It gets harder to read when your json has with lots fields and components. For proper formatting, the library has the nice pretty formatting mode which prints the each field in one line To use this, you need to create Gson instance with GsonBuilder#setPrettyPrinting() as shown in following listing Following is the our class, whose object to converted to its json representation, class Developer {private String name; private String classz; List<String> languagesKnown; public Developer() { name = "ajduke"; languagesKnown = new ArrayList<>(); languagesKnown.add("Java"); languagesKnown.add("Scala"); languagesKnown.add("Ruby"); } } Note line no. 7 for use, Gson gson = new Gson(); String json = gson.toJson(new Developer()); System.out.println("********* Compact mode ***********"); System.out.println(json); GsonBuilder gsonBuilder = new GsonBuilder(); Gson prettyGson = gsonBuilder.setPrettyPrinting().create(); json = prettyGson.toJson(new Developer()); System.out.println("\n ******* Pretty formatting *********"); System.out.println(json); and output for above is ********* Compact mode *********** {"name":"ajduke","languagesKnown":["Java","Scala","Ruby"]}******* Pretty formatting ********* { "name": "ajduke", "languagesKnown": [ "Java", "Scala", "Ruby" ] } Including null fields in JSON output By default, in conversion null value fields are ignored while converting to Json format. This is just to make output clear and compact. But we can override this behaviour by creating the Gson instance with GsonBuilder #serializeNulls(). Following listing shows class Developer {private String name; private String classz; List<String> languagesKnown; public Developer() { name = "ajduke"; classz= "Developer"; languagesKnown = new ArrayList<>(); languagesKnown.add("Java"); languagesKnown.add("Scala"); languagesKnown.add("Ruby"); } } Note line no .11 for use, Gson gson = new Gson();System.out.println("Default behaviour "); GsonBuilder gsonBuilder = new GsonBuilder();Gson prettyGson = gsonBuilder.setPrettyPrinting().create(); String json = prettyGson.toJson(new Developer()); System.out.println(json);System.out.println("Including the nulls "); Gson includeNullsGson = gsonBuilder.serializeNulls().create(); String json2 = includeNullsGson.toJson(new Developer()); System.out.println(json2); Note the output for the above, classz is included Default behaviour { "name": "ajduke", "languagesKnown": [ "Java", "Scala", "Ruby" ] } Including the nulls { "name": "ajduke", "classz": null, "languagesKnown": [ "Java", "Scala", "Ruby" ] }Including the custom name By default, json output field names are the same as the class field names. Gson provides the way to change this with use of an annotation. For this we need to use @serilizedName annotation provided for the field which we need to change the name in output json. So, our Developer class we change the name to firstName class Developer { @SerializedName("firstName") private String name; private String classz; List<String> languagesKnown; public Developer() { name = "ajduke"; languagesKnown = new ArrayList<>(); languagesKnown.add("Java"); languagesKnown.add("Scala"); languagesKnown.add("Ruby"); } } and if we run following then Gson gson = new Gson(); String json = gson.toJson(new Developer()); System.out.println("********* Compact mode ***********"); System.out.println(json); GsonBuilder gsonBuilder = new GsonBuilder(); Gson prettyGson = gsonBuilder.setPrettyPrinting().create(); json = prettyGson.toJson(new Developer()); System.out.println("\n ******* Pretty formatting *********"); System.out.println(json); then in output we will have the “firstName” instead of “name” ********* Compact mode *********** {"firstName":"ajduke","languagesKnown":["Java","Scala","Ruby"]}******* Pretty formatting ********* { "firstName": "ajduke", "languagesKnown": [ "Java", "Scala", "Ruby" ] }Writing the JSON to/from stream Gson library has one overload to each method, toJson() and fromJson(), for writing and reading from streams respectively. In simple words, you can write json output to file, character, or  buffer and you can read from any Reader stream. Following listing shows, writing and reading JSON to/from File stream package in.ajduke.ap013; import java.io.FileWriter; import java.io.IOException; import java.util.ArrayList; import java.util.List; import com.google.gson.Gson; import com.google.gson.GsonBuilder; import com.google.gson.annotations.SerializedName; public class GsonEx { public static void main(String[] args) throws IOException { Gson gson = new Gson(); System.out.println("Writing JSON to file ...."); // using try with resources try (FileWriter writer = new FileWriter("d:/output.json")) { gson.toJson(new Developer(), writer); // writing to file } System.out.println("Reading from file.. "); FileReader fileReader = new FileReader("d:/output.json"); dev= gson.fromJson(fileReader, Developer.class); System.out.println(dev); } }class Developer { private String name; private String classz; List<String> languagesKnown; public Developer() { name = "ajduke"; languagesKnown = new ArrayList<>(); languagesKnown.add("Java"); languagesKnown.add("Scala"); languagesKnown.add("Ruby"); } @Override public String toString() { return "Developer [name=" + name + ", classz=" + classz + ", languagesKnown=" + languagesKnown + "]"; } }   Reference: Using Google GSON : Extra Goodies : Part I from our JCG partner Abhijeet Sutar at the ajduke’s blog blog. ...

Spring LDAP 2.0.0 is released

The Spring team is pleased to announce that Spring LDAP 2.0.0 is released and is now available from Maven Central and Bintray. Spring LDAP 2.0.0.RELEASE Released lists all changes in the new release. JIRA’s change logs contain a list of the changes too. The most basic features of 2.0.0 release are presented below:Spring LDAP now includes Spring Data Repository support as also QueryDSL support. Spring LDAP also includes Fluent LDAP query support, making it much more interesting to work with LDAP search. Now it’s much easier to configure Spring LDAP, since a custom XML namespace can be used. Spring LDAP core has been updated with Java 5 features, for example generics and varargs. The ODM (Object-Directory Mapping) functionality has been moved to core and new methods are provided in LdapOperations/LdapTemplate for automatic translation to/from ODM-annotated classes. The new release requires Java 6....

Project Student: Webservice Integration

This is part of Project Student. Other posts are Webservice Client With Jersey, Webservice Server with Jersey, Business Layer, Persistence with Spring Data and Sharding Integration Test Data. Earlier we successfully ran integration tests for both the persistence/business layer (using an embedded H2 database) and the REST server/client layers (using a Jetty server). It’s time to knit everything together. Fortunately we already have all of our code in place and tested. All we need to do now is create some configuration file magic.   LimitationsUser authentication – no effort has been made to authenticate users. Encryption – no effort has been made to encrypt communications.Container-Managed Datasource and JNDI Container-managed datasources have a bad reputation for many developers and I’m not sure why. Confusion with Container-Managed Persistence (CMP), perhaps? In any case the idea behind a container-managed datasource is simple. You don’t need to figure out how to maintain database connection parameters in a deployed system – no need to modify a deployed webapp (which is insecure) or read a file from the filesystem (which you may not be able to access), etc. You just hand the problem to the person maintaining the webserver/appserver and retrieve the value via JNDI. Tomcat and Jetty require manual configuration in an XML file. More advanced appservers like JBoss and GlassFish allow you to configure datasources via a nice GUI. It doesn’t really matter since you only have to do it once. Instructions for Tomcat 7 and Jetty. The key points for Tomcat are that the server library is under $CATALINA_HOME/lib and that might not be where you expect. E.g., in Ubuntu it’s /usr/share/tomcat7/lib, not /var/lib/tomcat7/lib. Second, if the META-INF/context.xml file isn’t picked up from the .war file you must place it under conf/Catalina/localhost/student-ws-webapp.xml (or whatever you have named your .war file). The latter overrides the former so it’s common to set up the .war file to run in the development environment and then override the configuration in the test and production environments. META-INF/context.xml (Tomcat) <?xml version="1.0" encoding="UTF-8"?> <Context><Resource name="jdbc/studentDS" auth="Container" type="javax.sql.DataSource" driverClassName="org.postgresql.Driver" url="jdbc:postgresql:student" username="student" password="student" maxActive="20" maxIdle="10" maxWait="-1" factory="org.apache.commons.dbcp.BasicDataSourceFactory" /></Context> It’s worth noting that it’s trivial to pass an encryption key via JNDI (as a java.lang.String). This has the same benefits as discussed earlier with regards to the need to modify a deployed webapp or access the server’s filesystem. (The implementation is a little more complex since you want your actual encryption key to require both the JNDI key AND a filesystem-based salt but this is easy to handle during initial deployment of the webapp.) JPA Configuration Our persistence.xml file is extremely minimal. We’ll typically want two persistence units, one for JTA transactions (production) and one for non-JTA transactions (development and testing). META-INF/persistence.xml <?xml version="1.0" encoding="UTF-8"?> <persistence xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd" version="1.0"><persistence-unit name="studentPU-local" transaction-type="RESOURCE_LOCAL"> <provider>org.hibernate.ejb.HibernatePersistence</provider> <non-jta-data-source>jdbc/studentDS</non-jta-data-source> </persistence-unit> </persistence> Web Configuration The web.xml file is nearly identical to the one used in integration testing. The key difference is that it pulls in a resource reference for the container-provided datasource. WEB-INF/web.xml <?xml version="1.0" encoding="UTF-8"?> <web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"><display-name>Project Student Webservice</display-name><context-param> <param-name>contextClass</param-name> <param-value>org.springframework.web.context.support.AnnotationConfigWebApplicationContext</param-value> </context-param> <context-param> <param-name>contextConfigLocation</param-name> <param-value> com.invariantproperties.sandbox.student.config.PersistenceJpaConfig com.invariantproperties.sandbox.student.config.BusinessApplicationContext com.invariantproperties.sandbox.student.webservice.config.RestApplicationContext </param-value> </context-param><listener> <listener-class> org.springframework.web.context.ContextLoaderListener </listener-class> </listener><servlet> <servlet-name>REST dispatcher</servlet-name> <servlet-class>com.sun.jersey.spi.spring.container.servlet.SpringServlet</servlet-class> <init-param> <param-name>spring.profiles.active</param-name> <param-value>test</param-value> </init-param> </servlet><servlet-mapping> <servlet-name>REST dispatcher</servlet-name> <url-pattern>/rest/*</url-pattern> </servlet-mapping><resource-ref> <description>Student Datasource</description> <res-ref-name>jdbc/studentDS</res-ref-name> <res-type>javax.sql.DataSource</res-type> <res-auth>Container</res-auth> </resource-ref> </web-app> Spring Configuration The Spring configuration for the persistence layer is dramatically different from before because of two changes. Stylistically, we can use a standard configuration file instead of a configuration class since we no longer have to deal with an embedded database. The more important change is that we’ve moved all of the datasource configuration to the container and we can eliminate it from our spring configuration. We need to point to the right datasource and persistence unit and that’s about it! applicationContext-dao.xml <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:context="http://www.springframework.org/schema/context" xmlns:tx="http://www.springframework.org/schema/tx" xmlns:jpa="http://www.springframework.org/schema/data/jpa" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jee="http://www.springframework.org/schema/jee" xsi:schemaLocation="http://www.springframework.org/schema/beanshttp://www.springframework.org/schema/beans/spring-beans-3.2.xsdhttp://www.springframework.org/schema/contexthttp://www.springframework.org/schema/context/spring-context-3.2.xsdhttp://www.springframework.org/schema/txhttp://www.springframework.org/schema/tx/spring-tx-3.2.xsdhttp://www.springframework.org/schema/data/jpahttp://www.springframework.org/schema/data/jpa/spring-jpa-1.3.xsdhttp://www.springframework.org/schema/jeehttp://www.springframework.org/schema/jee/spring-jee-3.0.xsd"><context:property-placeholder location="WEB-INF/database.properties" /><context:annotation-config /><!-- we use container-based datasource --> <jee:jndi-lookup id="dataSource" jndi-name="${persistence.unit.dataSource}" expected-type="javax.sql.DataSource" /><bean name="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean"> <property name="dataSource" ref="dataSource" /> <property name="persistenceUnitName" value="${persistence.unit.name}" /> <property name="packagesToScan" value="${entitymanager.packages.to.scan}" /> <property name="jpaVendorAdapter"> <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter" /> </property> </bean><bean name="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager" /><bean name="exceptionTranslation" class="org.springframework.dao.annotation.PersistenceExceptionTranslationPostProcessor" /><tx:annotation-driven transaction-manager="transactionManager" proxy-target-class="false" /></beans> Our properties file just contains the name of the datasource, the persistence unit name, and the list of packages to scan that contain JPA annotations. WEB-INF/database.properties # jpa configuration entitymanager.packages.to.scan=com.invariantproperties.sandbox.student.domain persistence.unit.dataSource=java:comp/env/jdbc/studentDS persistence.unit.name=studentPU-local Database Schema and Security Finally the integration tests can use Hibernate auto-creation but we should always explicitly maintain our schemas in a development and production environment. This allows us to maintain infrastructure values in addition to avoiding potential problems from automated tools acting in an unexpected manner. It’s important to write and test both upgrade and downgrade scripts before going into production. We always need a way to recover gracefully if there’s a problem. More importantly our database schema should always be owned by a different user than the webapp. E.g., the webapp may use ‘student-user’ for the tables created by ‘student-owner’. This will prevent an attacker using SQL injection from deleting or modifying tables. -- -- for security this must run as student-owner, not student-user! ---- -- create an idempotent stored procedure that creates the initial database schema. -- create or replace function create_schema_0_0_2() returns void as $$ declare schema_version_rec record; schema_count int; begin create table if not exists schema_version ( schema_version varchar(20) not null );select count(*) into schema_count from schema_version;case schema_count when 0 then -- we just created table insert into schema_version(schema_version) values('0.0.2'); when 1 then -- this is 'create' so we only need to make sure it's current version -- normally we accept either current version or immediately prior version. select * into strict schema_version_rec from schema_version; if schema_version_rec.schema_version '0.0.2' then raise notice 'Unwilling to run updates - check prior version'; exit; end if; else raise notice 'Bad database - more than one schema versions defined!'; exit; end case;-- create tables!create table if not exists test_run ( test_run_pkey int primary key, uuid varchar(40) unique not null, creation_date timestamp not null, name varchar(80) not null, test_date timestamp not null, username varchar(40) not null );create table if not exists classroom ( classroom_pkey int primary key, uuid varchar(40) unique not null, creation_date timestamp not null, test_run_pkey int references test_run(test_run_pkey), name varchar(80) not null );create table if not exists course ( course_pkey int primary key, uuid varchar(40) unique not null, creation_date timestamp not null, test_run_pkey int references test_run(test_run_pkey), name varchar(80) not null );create table if not exists instructor ( instructor_pkey int primary key, uuid varchar(40) unique not null, creation_date timestamp not null, test_run_pkey int references test_run(test_run_pkey), name varchar(80) not null, email varchar(200) unique not null );create table if not exists section ( section_pkey int primary key, uuid varchar(40) unique not null, creation_date timestamp not null, test_run_pkey int references test_run(test_run_pkey), name varchar(80) not null );create table if not exists student ( student_pkey int primary key, uuid varchar(40) unique not null, creation_date timestamp not null, test_run_pkey int references test_run(test_run_pkey), name varchar(80) not null, email varchar(200) unique not null );create table if not exists term ( term_pkey int primary key, uuid varchar(40) unique not null, creation_date timestamp not null, test_run_pkey int references test_run(test_run_pkey), name varchar(80) not null );-- correction: need to define this! create sequence hibernate_sequence;-- make sure nobody can truncate our tables revoke truncate on classroom, course, instructor, section, student, term, test_run from public;-- grant CRUD privileges to student-user. grant select, insert, update, delete on classroom, course, instructor, section, student, term, test_run to student;grant usage on hibernate_sequence to student;return; end; $$ language plpgsql;-- create database schema select create_schema_0_0_2() is null;-- clean up drop function create_schema_0_0_2(); Integration Testing We can reuse the integration tests from the webservice service but I haven’t copied them to this project since 1) I hate duplicating code and it would be better to pull the integration tests into a separate project used by both server and webapp and 2) it’s easy to configure Jetty to supply JNDI values but for some reason the documented class is throwing an exception and it’s not important enough (at this time) to spend more than a few hours researching. Source CodeThe source code is available at http://code.google.com/p/invariant-properties-blog/source/browse/student.Correction The schema provided overlooked the ‘hibernate_sequence’ required to create new objects. It must be defined and readable by the ‘student’ user.   Reference: Project Student: Webservice Integration from our JCG partner Bear Giles at the Invariant Properties blog. ...

Java vs. Native Agents – The Powerful Things They Do

What you should know before installing an agent, and how it affects your code When building a scalable server-side application we spend considerable amount of time thinking about how we’ll monitor, operate and update our code in production. A new breed of tools have evolved to help Java and Scala developers do just that. Many of them are built on one of the most powerful ways in which external code can integrate with the JVM at runtime – Java agents. Agents are OS native or Java libraries (we’ll describe the differences below) to which the JVM provides capabilities that aren’t available to normal application code. To get a sense of how fundamental these are, let’s look at just a few of the tools we use that rely on them -Profilers use a Java agent to modify target framework’s code to inject new code that collects performance metrics. This includes both standalone or hosted services such as NewRelic or YourKit.The Play framework V1 uses a Java agent to enable hot-swapping of classes at run-time.JRebel took it to the next level by building a technology which provides smooth hot-swapping of classes at runtime without requiring JVM restarts.At Takipi we leverage low-level capabilities provided by the JVM to native agents to show the actual source code and variables values which cause errors.What can agents do? As I alluded above, there are two kinds of agents – Java and native. While both are loaded into JVM in almost the same manner (using a special JVM startup argument), they’re almost completely different in how they’re built and what they’re meant to do. Let’s look at the two  - Java agentsJava agents are .jar files that define a special premain static function which will be invoked by the JVM before the application’s main function is invoked. The magical part comes in with the Instrumentation object, which is passed as an argument to this function by the host JVM. By holding on to this object the agent’s code (which otherwise behaves as any Java code loaded by the root class loader) can do some really powerful things. public static void premain(String agentArgs, Instrumentation inst) { myInst = inst; //grab a reference to the inst object for use later } What they do. The most powerful capability given to the agent is the ability to dynamically rewrite the contents of a target class’ method at run-time class (field structures are immutable). This process, known as byte code instrumentation, enables the agent to essentially rewrite the contents of a method while the code is running. Some examples include adding calls to specific methods to profile performance (e.g. endtime – starttime) or record parameters values (e.g. the URL passed to a servlet). Another example would be to reload a new version of a class without restarting the JVM, as done by JRebel. How it’s done. For an agent to modify the code or a loaded class it essentially triggers a process of a class reload by the JVM, where the class’s bytecode is replaced with a new version. This requires that the agent be able to provide the JVM with new bytecode that is verifiable (i.e. conforms to the JVM specification). Unfortunately, generating correct bytecode at runtime isn’t very simple – there are a lot of requirements and edge-cases. For this agents usually use a library for reading and writing bytecode that enables them to load an existing class’s bytecode into a DOM-like structure, modify it by adding things like profiling calls, and then save the DOM back to raw bytecode. ASM is a popular library for this. So popular, that it’s in fact used by some of Sun’s internal code to parse bytecode in Java. Native agentsNative agents are completely different beasts. If you thought Java agents can let you do cool things, hold on to your socks, because native agents operate on a whole different level. Native agents are not written in Java, but mostly in C++, and are not subjected to the rules and restrictions under which normal Java code operates. Not only that, they’re provided with an extremely powerful set of capabilities called the JVM Tooling Interface (JVMTI). What they do. This set of of APIs exposed by jvmti.h essentially enables a C++ library dynamically loaded by the JVM to obtain an extremely high level of visibility into the real- time working of the JVM. This spans across a wide field of areas, including GC, locking, code manipulation, synchronization, thread management, compilation debugging and much more. The JVM TI has been designed to make the JVM as transparent as possible while still maintaining the design flexibility to allow JVM vendors to provide different underlying implementations. This set of APIs is very wide, containing literally hundreds of callbacks and functions into the JVM.  You can use these to do extremely powerful things that Java agents can’t do such as writing you own debugger, or building low-level, real time error analysis tools (which is what Takipi is). For example, here’s a callback that the JVMTI provides to the agent, so that whenever an exception is thrown anywhere inside the JVM, the agent will be receive the bytecode location in which the exception was thrown, the owner thread, the exception object, and if / where it will be caught. Powerful stuff indeed. void JNICALL ExceptionCallback(jvmtiEnv *jvmti, JNIEnv *jni, jthread thread, jmethodID method, jlocation location, jobject exception, jmethodID catch_method, jlocation catch_location) The downside. If everything I described sounds absolutely peachy, you can ask why aren’t all agents written as native agents? There are few of reasons to be aware of  so here they are (in no particular order). Complexity. The first is that the JVMTI API is very complex, with a lot of small moving wheels.For the most part, if you’re not wiring an agent that requires very low-level capabilities you’re fine with the Java agent API, which is more straightforward and can help you get the job done more quickly. Portability. Since native agents are written and compiled as native libraries (.so / .dll) they need to be compiled and tested across any number of operating systems you want to support. If you look at Windows, OSX and the different flavours Linux comes in, that can translate to a lot of work. Compare that with Java agents which are executed as Java code by the JVM, and therefore are inherently portable by design. Bytecode manipulation. Since native agents are usually written in C++, it means they can’t use tried and true Java bytecode manipulation libraries (such as ASM) directly without going back into the JVM using JNI, which does take some of the fun out. Stability. The JVM provides strong safeguards to prevents code from doing the kinds of things that will cause an angry OS to terminate the process. Memory access violations that would under normal circumstances cause a SIGSEV and crash the process, get wrapped us a nice NullPointerException. Since native agents run at the same level of the JVM (vs. Java agents whose code is executed by it),any mistakes they make can potentially terminate the JVM. Hope this helps highlight some of the differences between the two kinds. It’s good to know what are agents and how they’re built, as even if you never end up writing one, you may be relying on one or more of them to power your application today.   Reference: Java vs. Native Agents – The Powerful Things They Do from our JCG partner Iris Shoor at the Takipi blog. ...

Android Service Tutorial

In this post, we want to talk about Android Service. This is a key component in developing Android app. Differently from Activity, Service in Android runs in background, they don’t have an interface and have a life-cycle very different from Activities. Using Service we can implement some background operation, for example we can suppose we want to load a web page from a remote server. Using a Service we can implement multitask in Android. Service Overview We already know that Android Activity can be started, stopped, destroyed if the system resources become too low and maybe can be recreated, a Service are designed to have a longer life. A Service in Android can be started from an Activity, from a Broadcast receiver and other services too. We have to notice that using Service we don’t create automatically new threads, so if we implement a simple logic inside our Service, that doesn’t require long time processing we don’t need to run it a separate thread, but if we have to implement complex logic, with long time processing, we have to take care of creating a new thread, otherwise the service runs on the main thread and it could cause ANR problem. In Android a Service is used for two main reason:Implement multi-task Enable Inter-Process-Communication (IPC)A typical example of the first case is an app that required to download data from a remote server, in this case we can have Activity that interacts with a user and starts a service that accomplishes the work in background while the user can use the app and maybe when the service finishes sends a message to the user. In the second case, we want to “share” some common functions so that different app can re-use them. For example, we can suppose we have a service that sends an email and we want to share this service among several apps so that we don’t have to rewrite the same code. In this case we can use IPC so that the service exposes a “remote” interface that can be called by other app. In this post, we cover the first case, in this case we have a local service, local means that the service can be seen just inside our apk. Service basic Now we know more about Service, we want to create it. In Android to create a Service we have to extend Service class. public class TestService extends Service {@Override public IBinder onBind(Intent arg0) { return null; }} As we can see, we have to implement only one method called onBind. In our case, we are using local service, so this method should return null. As we mentioned before, a Service has its lifecycle and we can override some callback methods so that we can handle its different states: public class TestService extends Service {@Override public void onCreate() { super.onCreate(); }@Override public void onDestroy() { super.onDestroy(); }@Override public int onStartCommand(Intent intent, int flags, int startId) { return super.onStartCommand(intent, flags, startId); }@Override public IBinder onBind(Intent arg0) { return null; }} The first method onCreate is called only one time when the Service has to created. If the Service is already running this method won’t be called. We don’t call it directly but it is the OS that calls it. OnStartCommand is the most important method because it is called when we required to start the Service. In this method we have the Intent passed at time we run the Service, in this way we can exchange some information with the Service. In this method, we implement our logic that can be execute directly inside this method if it isn’t time expensive otherwise we can create a thread. As you can see this method requires we return an Integer as result. This integer represents how the Service should be handled by the OS:START_STICKY : Using this return value, if the OS kills our Service it will recreate it but the Intent that was sent to the Service isn’t redelivered. In this way the Service is always running START_NOT_STICKY: If the SO kills the Service it won’t recreate it until the client calls explicitly onStart command  START_REDELIVER_INTENT: It is similar to the START_STICKY and in this case the Intent will be redelivered to the service.OnDestroy is the method called by the OS when the Service will be destroyed. Once we have our service class, we have to declare it in the Manifest.xml so that we can use it: <service android:name=".TestService" android:enabled="true"/> Start and Stop Service As we know a Service has to be started and eventually stopped so that it can accomplish its task. We can suppose to start it from an activity and we could pass to the service some information using Intent. We can suppose that our Activity has two buttons one to start and one to stop the Service: btnStart.setOnClickListener(new View.OnClickListener() {@Override public void onClick(View v) { Intent i = new Intent(MainActivity.this, TestService.class); i.putExtra("name", "SurvivingwithAndroid"); MainActivity.this.startService(i); } });btnStop.setOnClickListener(new View.OnClickListener() {@Override public void onClick(View v) { Intent i = new Intent(MainActivity.this, TestService.class); MainActivity.this.stopService(i); } }); In the code above at line 5, we create an Intent passing the class name that handles our Service, moreover we set some params like name and then we start the Service at line 7. In the same way, at line 17 we stop the service.Clicking on Start button we get in the Log:We can notice that the onCreate method is called because it is the first time we start the service, if we click again on start button the OS doesn’t call onCreate method. When we click on stop button the OS destroy the service. IntentService As we mentioned before a service runs on the main thread, so we have to be very careful when we implement some logic in this service. We have to consider that if this logic is a blocking operation or it requires long time to finish a ANR problem could occur. In this case we have to move our logic to a separate thread, meaning we have to create a thread in onStartCommand method and run it. There is another class called IntentService derived from Service that simplify our life. This class is useful when we don’t need to handle multiple requests at the same time. This class creates a worker thread to handle different requests. This class performs this operation:create a separate thread to handle the request create a request queue and pass one Intent at time create a default implementation of onStartCommand stop the service when all the requests are processedIf we want to create a IntentService we have to extend IntentService class instead of Service: public class TestIntentService extends IntentService {public TestIntentService() { super("TestIntentService"); }@Override protected void onHandleIntent(Intent intent) {}} In this case we have only one method to implement called onHandleIntent. Here we implement out logic without caring if this is operation requires long time or not, because this method is called in a separate thread. Starting service automatically Many times we want to start our service automatically, for example at boot time. We know to start a Service we need a component to start it. How can we do it? Well, we could use a Broadcast receiver that starts our service. If for example we want to start it at the smartphone boot time, we create first a Broadcast receiver that listens to this event and then it starts the service. public class BootBroadcast extends BroadcastReceiver {@Override public void onReceive(Context ctx, Intent intent) { ctx.startService(new Intent(ctx, TestService.class));}} and in the Manifest.xml <receiver android:name=".BootBroadcast"> <intent-filter > <action android:name="android.intent.action.BOOT_COMPLETED"/> </intent-filter> </receiver>   Reference: Android Service Tutorial from our JCG partner Francesco Azzola at the Surviving w/ Android blog. ...

Why Your Boring Data Will Outlast Your Sexy New Technology

So you’re playing around with all those sexy new technologies, enjoying yourself, getting inspiration from state-of-the-art closure / lambda / monads and other concepts-du-jour… Now that I have your attention provoking a little anger / smirk / indifference, let’s think about the following. I’ve recently revisited a great article by Ken Downs written in 2010. There’s an excellent observation in the middle. [...] in fact there are two things that are truly real for a developer:The users, who create the paycheck, and The data, which those users seemed to think was supposed to be correct 100% of the time.He says this in the context of a sophisticated rant against ORMs. It can also be seen as a rant against any recent abstraction or database technology with a mixture of nostalgia and cynicism. But the essence of his article lies in another observation that I’ve made with so many software systems:Developers tend to get bored with “legacy” Systems tend to outlast their developers Data tends to outlast the systems operating on itHaving said so, we can quickly understand why Java has become the new COBOL to some, who are bored with its non-sexiness (specifically in the JEE corner). Yet, according to TIOBE, Java and C are still the most widely used programming languages in the market and many systems are thus implemented in Java. On the other hand, many inspiring new technologies have emerged in the last couple of years, also in the JVM field. A nice overview is given in ZeroTurnaround’s The Adventurous Developer’s Guide to JVM Languages. And all of that is fine to a certain extent. As Bruno Borges from Oracle recently and adequately put it: anything not mainstream has more odds to be “sexy” [than JSF] @tobrien anything not mainstream has more odds to be “sexy” @lukaseder @myfear— Bruno Borges (@brunoborges) December 28, 2013 Now, let’s map this observation back to a subsequent section of Ken’s article: [...] the application code suddenly becomes a go-between, the necessary appliance that gets data from the db to the user [...] and takes instructions back from the user and puts them in the database [...]. No matter how beautiful the code was, the user would only ever see the screen [...] and you only heard about it if it was wrong. Nobody cares about my code, nobody cares about yours. Think about an E-Banking system. None of your users really cares about the code you wrote to get it running. What they care about is their data (i.e. their money) and the fact that it is correct. This is true for many many systems where the business logic and UI layers can be easily replaced with fancy new technology, whereas the data stays around. In other words, you are free to choose whatever sexy new UI technology you like as long as it helps your users get access to their data. So what about sexy new database technology? Now, that’s an entirely different story, compared to sexy new UI technology. You might be considering some NoSQL-solution-du-jour to “persist” your data, because it’s so easy and because it costs so much less. Granted, the cost factor may seem very tempting at first. But have you considered the fact that:Systems tend to outlast their developers Data tends to outlast the systems operating on itOnce your data goes into that NoSQL store, it may stay there much longer than you had wanted it to. Your developers and architects (who originally chose this particular NoSQL solution) may have left long ago. Parts of your system may have been replaced, too, because now you’re doing everything in HTML5. JavaScript is the new UI technology. And all this time, you have “persisted” UI / user / domain model data in your database, from various systems, from various developers, through various programming paradigms. And then, you realise: What the hell have you built. http://t.co/Tq7ktgDWzD — Jeff Atwood (@codinghorror) June 18, 2013 We’re not saying that there aren’t some use-cases where NoSQL databases really provide you with a better solution than the “legacy” alternatives. Most specifically, graph databases solve a problem that no RDBMS has really solved well, yet. But consider one thing. You will have to migrate your data. Time and again. You will have to archive it. And maybe, migrate the archive. And maybe provide reports of the archive. And provide correct reports (which means: ACID!) And be transactional (which means: ACID!) And all the other things that people do with data. In fact, your system will grow like any other system ever did before and it will have high demands from your database. While some NoSQL databases have started to get some decent traction, in a way that it is safe to say their vendors might still be around in 5-10 years, when the systems will have been replaced by developers who have replaced other developers. In other words, there is one more bullet to this list:Developers tend to get bored with “legacy” Systems tend to outlast their developers Data tends to outlast the systems operating on it Data might tend to outlast the vendors providing tools for operating the dataBeware of the fact that data will probably outlast your sexy new technology. So, choose wisely when thinking about sexy new technology to operate your data.   Reference: Why Your Boring Data Will Outlast Your Sexy New Technology from our JCG partner Lukas Eder at the JAVA, SQL, AND JOOQ blog. ...

Zabbix – Low Level Discovery of cores, CPUs and Hard Disk

Zabbix SSD Status, configured with Low Level Discovery At home, I’m using Zabbix to monitor my servers, it has plenty of interesting features and can be extended a lot by using User Parameter. In this post, I’m gonna talk about Low Level Discovery (LLD). If you are only interested in the final result, go the Conclusion section, you can download my template containing all the rules! Low Level Discovery (LLD) LLD is a feature to automatically discover some properties of the monitored host and create items, triggers and graphs. By default, Zabbix support three types of item discovery:Mounted filesystems Network interface SNMP’s OIDsThe first two are very useful, since they will give you by default, for instance, the free space of each mounted file system or the bandwith going in and out of each network interface. As I only monitor Linux servers, I don’t use the last one, but it will eventually interest other people. Another very interesting thing about this feature is that you can extend it by discovering more items. In this article, I will show how to discover CPUs, CPU Cores and Hard Disk. The most important part of custom discovery is to create a script on the monitored machines that can “discover” something. It can be any executable, the only thing important is that it outputs data in the correct format. I have to say that the format is quite ugly, but that is probably not very important.  Here is the output of my hard disk discovery script: { "data":[ {"{#DISKNAME}":"/dev/sda","{#SHORTDISKNAME}":"sda"}, {"{#DISKNAME}":"/dev/sdb","{#SHORTDISKNAME}":"sdb"}, {"{#DISKNAME}":"/dev/sdc","{#SHORTDISKNAME}":"sdc"}, {"{#DISKNAME}":"/dev/sdd","{#SHORTDISKNAME}":"sdd"}, {"{#DISKNAME}":"/dev/sde","{#SHORTDISKNAME}":"sde"}, {"{#DISKNAME}":"/dev/sdf","{#SHORTDISKNAME}":"sdf"}, {"{#DISKNAME}":"/dev/sdg","{#SHORTDISKNAME}":"sdg"}, ] } You can have as many keys for each discovered items, but the format must remains the same. In the item, trigger and graph prototypes, you will then use {#DISKNAME} or {#SHORTDISKNAME} to use the discovered values. Once you have created your scripts, you have to register it in the zabbix configuration as a user parameter. For instance, if you use the zabbix daemon, you need these lines in /etc/zabbix/zabbix_agentd.conf: EnableRemoteCommands=1 ... UnsafeUserParameters=1 ... UserParameter=discovery.hard_disk,/scripts/discover_hdd.sh Now, when you will create the discovery rule, you can use discovery.hard_disk as the key. A discovery rule in itself is useful without prototypes, you can create three types of prototypes:Item Prototype: This will create a new item for each discovered entity Trigger Prototype: This will create a new trigger for each discovered entity. Graph Prototype: This will create a graph for each discovered entity.The most useful are by far the item and trigger prototypes. The biggest problem with graphs is that you cannot create an aggregate graph of each discovered items. For instance, if you record the temperature of your CPU cores, you cannot automatically create a graph with the temperature of each discovered cores. For that, you have to create the graph in each host. Which makes, imho, graph prototypes pretty useless. Anyway… In the next section, I’ll show how I have created discovery rules for Hard Disk, CPU and CPU cores. Discover Hard Disk The discovery script is really simple: #!/bin/bashdisks=`ls -l /dev/sd* | awk '{print $NF}' | sed 's/[0-9]//g' | uniq`echo "{" echo "\"data\":["for disk in $disks do echo " {\"{#DISKNAME}\":\"$disk\",\"{#SHORTDISKNAME}\":\"${disk:5}\"}," doneecho "]" echo "}" It just lists all the /dev/sdX devices, remove the partition number and remove the duplicates, to have only the hard disk at the end. I’ve created several item prototypes for each hard disk. Here are some examples using S.M.A.R.T. (you can download the template with all the items in the Conclusion section):Raw Read Error Rate Spin Up Time SSD Life Left Temperature …You may notice that some of them only make sense for SSD (SSD Life Left) and some others do not make any sense for SSD (Spin Up Time). This is not a problem since they will just be marked as Not Supported by Zabbix. All these datas are collected using the smartctl utility. I’ve also created some trigger to indicate the coming failure of an hard disk:SSD Life Left too low Reallocated Sector Count too low …I’ve just used the threshold reported by smartctl, they may be different from one disk manufacturers to another. I don’t put a lot of faith on these values, since disk generally fail before going to threshold, but it could be a good indicator anyway. Discover CPUs Here is the script to discover CPUs: #!/bin/bashcpus=`lscpu | grep "CPU(s):" | head -1 | awk '{print $NF}'` cpus=$(($cpus-1))echo "{" echo "\"data\":["for cpu in $(seq 0 $cpus) do echo " {\"{#CPUID}\":\"$cpu\"}," doneecho "]" echo "}" It just uses lscpu and parses its output to find the number of CPU and then create an entry for each CPUs. I just have one item for each CPU: The CPU Utilization. I haven’t created any trigger here. Discover CPU Cores Just before, we discovered the CPUs, but it is also interesting to discover the cores. If you don’t have Hyperthreading, the result will be the same. It is especially interesting to get the temperature of each core. Here is the script: #!/bin/bashcores=`lscpu | grep "Core(s) per socket:" | awk '{print $NF}'` cores=$(($cores-1))echo "{" echo "\"data\":["for core in $(seq 0 $cores) do echo " {\"{#COREID}\":\"$core\"}," doneecho "]" echo "}" It works in the same way as the previous script. I’ve only created one item prototype, to get the temperature of each core with lm_sensors. Wrap-Up Here are all the UserParameter necessary to make the discovery and the items works: ### System Temperature ### UserParameter=system.temperature.core[*],sensors|grep Core\ $1 |cut -d "(" -f 1|cut -d "+" -f 2|cut -c 1-4### DISK I/O### UserParameter=custom.vfs.dev.read.ops[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$4}' UserParameter=custom.vfs.dev.read.ms[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$7}' UserParameter=custom.vfs.dev.write.ops[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$8}' UserParameter=custom.vfs.dev.write.ms[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$11}' UserParameter=custom.vfs.dev.io.active[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$12}' UserParameter=custom.vfs.dev.io.ms[*],cat /proc/diskstats | egrep $1 | head -1 y| awk '{print $$13}' UserParameter=custom.vfs.dev.read.sectors[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$6}' UserParameter=custom.vfs.dev.write.sectors[*],cat /proc/diskstats | egrep $1 | head -1 | awk '{print $$10}' UserParameter=system.smartd_raw[*],sudo smartctl -A $1| egrep $2| tail -1| xargs| awk '{print $$10}' UserParameter=system.smartd_value[*],sudo smartctl -A $1| egrep $2| tail -1| xargs| awk '{print $$4}'### Discovery ### UserParameter=discovery.hard_disk,/scripts/discover_hdd.sh UserParameter=discovery.cpus,/scripts/discover_cpus.sh UserParameter=discovery.cores,/scripts/discover_cores.sh (it must be set in zabbix_agentd.conf) You also need to give zabbix the right to use sudo with smartctl. For that, you have to edit your /etc/sudoers file and add this line: ALL ALL=(ALL)NOPASSWD: /usr/sbin/smartctl Conclusion and Download I hope that this helps some people to use Low Level Discovery in their Zabbix Monitoring Installation. LLD eases a lot the creation of multiple items discovery for hosts with different hardware or configuration. However, it has some problems for which I have not yet found a proper solution. First, you have to duplicate the client scripts on each host (or at least have them on a share available from each of them). Then, the configuration of each agent is also duplicated in the configuration of each host. The biggest problem I think is the fact that you cannot automatically create graph with the generated items of each discovered entities. For instance, I had to create a CPU Temperature graph in each of my host. If you have few hosts, like many, it is acceptable, but if you have hundreds of hosts, you just don’t do it. All the scripts and the template export file are available in the zabbix-lld repository. For everything to work, you need the lscpu, lm_sensors and smartmontools utilities. If you have any question or if something doesn’t work (I don’t offer any guarantee, but it should work on most recent Linux machines), don’t hesitate to comment on this post.   Reference: Zabbix – Low Level Discovery of cores, CPUs and Hard Disk from our JCG partner Baptiste Wicht at the @Blog(“Baptiste Wicht”) blog. ...

Mirror Mirror – Using Reflection To Look Inside The JVM at Run-time

Developers: Takipi tells you when new code breaks in production –  Learn more We’re all used to employing reflection in our everyday work, either directly, or through frameworks that leverage it. Its a main aspect of Java and Scala programming that enables the libraries we use to interact with our code without hard-coded knowledge of it. But our use of reflection is only limited to Java and Scala code that runs inside the JVM. What if we could use reflection to look not only into our code at run-time, but into the JVM’s code as well? When we began building Takipi, we looked for a way to efficiently analyze JVM heap memory to enable some low-level optimizations, such as scanning the address space of a managed heap block. We came across many interesting tools and capabilities to examine various aspects of the JVM state, and one of them does just that. It’s one of Java’s strongest and most low-level debugging tools – The Java Serviceability Agent. This powerful tool comes with the HotSpot JDK and enables us to see not only Java objects inside the heap, but look into the internal C++ objects comprising the JVM itself, and that’s where the real magic begins. Reflection ingredients. When dealing with any form of reflection to dynamically inspect and modify objects at runtime, two essential ingredients are required. The first one is a reference (or address) to the object you want to inspect. The second one is a description of the object’s structure, which includes the offsets in which its fields reside and their type information. If dynamic method invocation is supported, the structure would also contain a reference to the class’s method table (e.g. vtable) along with the parameters each one expects. Java reflection itself is pretty straightforward. You obtain a reference to a target object just like you would with any other. Its field and method structures are available to you via the universal Object.getClass method (originally loaded from the class’s bytecode). The real question is how do you reflect the JVM itself? The keys to the castle. Wonderfully enough, the JVM exposes its internal type system through a set of publicly exported symbols. These symbols provide the Serviceability agent (or anyone else for that matter) with access to the structures and addresses of the internal JVM class system. Through these one can inspect almost all aspects of the internal workings of the JVM at the lowest level, including things like raw heap addresses, thread/stack addresses and internal compiler states. Reflection in action. To get a sense of the possibilities, you can see some of these capabilities in action by launching the Serviceability Agent’s HotSpot Debugger UI. You can do this by launching sa-jdi.jar with sun.jvm.hotspot.HSDB as the main class argument. The capabilities you’ll see are the same ones that help power some of the JVM’s most powerful debugging tools such as jmap, jinfo and jstack.* HSDB and some of the extremely low-level inspection capabilities it offers into a target JVM. How it’s done. Let’s take a closer look to understand how these capabilities are actually provided by the JVM. The cornerstone for this approach lies with the gHotSpotVMStructs struct which is publicly exported by the jvm library. This struct exposes both the internal JVM type system as well as the addresses of the root objects from which we can begin reflecting. This symbol can be accessed just like you would dynamically link with any publicly exported OS library symbol via JNI or JNA. The question then becomes how do you parse the data in the address exposed by the gHotSpotVMStructs symbol? As you can see in the table below, the JVM exposes not only the address of its type system and root addresses, but also additional symbols and values that provide you with the values needed to parse the data. These include the class descriptors and binary offsets in which every field within a class is located.* A dependency walker screenshot of the symbols exposed by jvm.dll The manifest. The gHotSpotVMStructs structure points to a list of classes and their fields. Each class provides a list of fields. For each field the structures provide its name, type and whether its a static or non-static field. If it’s a static field the structure would also provide access to its value. In case of a static object type field, the structure would provide the address of the target object. This address serves as a root from which we can begin reflecting a specific component of the internal JVM system. This includes things like compiler, threading or collected heap systems. You can checkout the actual algorithm used by the Serviceability agent to parse the structure in the Hotspot JDK code here. Practical examples. Now that we’ve got a broad sense of what these capabilities can do, let’s take a look at some concrete examples of the types of data exposed by this interface. The folks who built the SA agent went through a lot of trouble to create Java wrappers around most of the classes provided by the gHotSpotVMStructs table. These provide a very clean and simple API to access most parts of the internal system in a way that is both type safe and hides most of the binary work required to access and parse the data. To give you a sense of some of the powerful capabilities exposed by this API, here are some references to the low-level classes it provides - VM is the singleton class which exposes many of the JVM’s internal systems such as the thread system, memory management and collection capabilities. It serves as an entry point into many of the JVM subsystems and is a good place to start when exploring this API. JavaThread gives you an inside look at how the JVM sees a Java thread from the inside, with deep information into frames locations and types (compiled, interpreted, native…) as well as actual native stack and CPU register information. CollectedHeap lets you explore the raw contents of the collected heap. Since HotSpot contains multiple GC implementations, this is an abstract class from which concrete implementations such as ParallelScavengeHeap inherit. Each provides a set of memory regions containing the actual addresses in which Java objects reside. As you look at the implementation of each class you’ll see it’s essentially just a hard coded wrapper using the reflection-like API to look into the JVM’s memory. Reflection in C++. Each of these Java wrappers is designed as an almost complete mirror of an internal C++ class within the JVM. As we know C++ doesn’t have a native reflection capability, which raises the question of how that bridge is created. The answer lies in something very unique which the JVM developers did. Through a series of C++ macros, and a lot of painstaking work, the HotSpot team manually mapped and loaded the field structures of dozens of internal C++ classes into the global gHotSpotVMStructs. This process is what makes them available for reflection from the outside. The actual field offset values and layouts are generated at the JVM compile time, helping to ensure the exported structures are compatible with the JVM’s target OS. Out-of-process connections. There’s one more powerful aspect to the Serviceability agent that’s worth taking a look at. One of the coolest capabilities the SA framework provides is the ability to reflect an external live JVM from out-of-process. This is done by attaching the Serviceability agent to the target JVM as an OS level debugger. Since this is OS dependent, for Linux the SA agent framework will leverage a gdb debugger connection. For Windows it will use winDbg (which means Windows Debugging Tools will be needed). The debugger framework is extensible, which means one could use another debugger by extending the abstract DebuggerBase class. Once a debugger connection is made, the return address value of gHotSpotVMStruct is passed back to the debugger process which can (by virtue of the OS) begin inspecting and even modifying the internal object system of the target JVM. This is exactly how HSDB lets you connect and debug the a target JVM, both the Java and JVM code.* HSDB’s interface exposes the SA agent’s capability of reflecting a target JVM process I hope this piqued your interest. From my own personal perspective, this architecture is one of my most favorite pieces of the JVM. Its elegance and openness are in my view definitely things to marvel at. It was also super helpful to us when we were building some of Takipi’s real-time encoding parts, so a big tip of the hat to the good folks who designed it.   Reference: Mirror Mirror – Using Reflection To Look Inside The JVM at Run-time from our JCG partner Tal Weiss at the Takipi blog. ...
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

15,153 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books