Featured FREE Whitepapers

What's New Here?


Knowledge Representation and Reasoning with Graph Databases

A graph database and its ecosystem of technologies can yield elegant, efficient solutions to problems in knowledge representation and reasoning. To get a taste of this argument, we must first understand what a graph is. A graph is a data structure. There are numerous types of graph data structures, but for the purpose of this post, we will focus on a type that has come to be known as a property graph. A property graph denotes vertices (nodes, dots) and edges (arcs, lines). Edges in a property graph are directed and labeled/typed (e.g. “marko knows peter”). Both vertices and edges (known generally as elements) can have any number of key/value pairs associated with them. These key/value pairs are called properties. From this foundational structure, a suite of questions can be answered and problems solved.   Object ModelingThe property graph data structure is nearly identical in form to the object graphs of object oriented programming. Take a collection of objects, remove their methods, and you are left with a property graph. An object’s fields are either primitive and in which cases serve as properties or they are complex and in which case serve as references to other objects. For example, in Java:   class Person { String name; Integer age; Collection<Person> knows; } The name and age properties are vertex properties of the particular person instance and the knows property refer to knows-labeled edges to other people. Emil Eifrem of Neo Technology espouses the view that property graphs are “whiteboard friendly” as they are aligned with the semantics of modern object oriented languages and the diagramming techniques used by developers. A testament to this idea is the jo4neo project by Taylor Cowan. With jo4neo, Java annotations are elegantly used to allow for the backing of a Java object graph by the Neo4j graph database. Beyond the technological benefits, the human mind tends to think in terms of objects and their relations. Thus, graphs may be considered “human brain friendly” as well. Given an object graph, questions can be answered about the domain. In the graph traversal DSL known as Gremlin, we can ask questions of the object graph: // Who does Marko know? marko.outE('knows').inV // What are the names of the people that Marko knows? marko.outE('knows').inV.name // What are the names and ages of the people that Marko knows? marko.outE('knows').inV.emit{[it.name, it.age]} // Who does Marko know that are 30+ years old? marko.outE('knows').inV{it.age > 30} Concept ModelingFrom the instances that compose a model, there may exist abstract concepts. For example, while there may be book instances, there may also be categories for which those books fall–e.g. science fiction, technical, romance, etc. The graph is a flexible structure in that it allows one to express that something is related to something else in some way. These somethings may be real or ethereal. As such, ontological concepts can be represented along with their instances and queried appropriately to answer questions.       // What are the parent categories of history? x = []; history.inE('subCategory').outV.aggregate(x).loop(3){!it.equals(literature)}; x // How many descendant categories does fiction have? c = 0; fiction.outE('subCategory').inV.foreach{c++}.loop(3){true}; c // Is romance at the same depth as history? c = 0; romance.inE('subCategory').outV.loop(2){c++; !it.equals(literature)}.outE('subCategory').inV.loop(2){c--; !it.equals(history)}; c == 0 Automated ReasoningFrom the explicit objects, their relationships, and their abstract categories, reasoning processes can be enacted. A tension that exists in graph modeling is what to make explicit (structure) and what to infer through traversal (process). The trade-off is between, like much of computing, space and time. If there exists an edge from a person to their coauthors, then its a single hop to get from that person to his or her coauthors. If, on the other hand, coauthors must be inferred through shared writings, then a multi-hop step is computed to determine coauthors. Reasoning is the process of making what is implicit explicit. A couple simple reasoning examples are presented below using Gremlin.   // Two people who wrote the same book/article/etc. are coauthors g.V{x = it}.outE('wrote').inV.inE('wrote').outV.except([x])[0].foreach{g.addEdge(null, x, it, 'hasCoauthor')} // People who write literature are authors author = g.addVertex(); author.type='role'; author.name='author' g.V.foreach{it.outE('wrote').inV[0].foreach{g.addEdge(null, it, author, 'hasRole')} >> -1} In the examples above, a full graph analysis is computed to determine all coauthors and author roles. However, nothing prevents the evaluation of local inference algorithms. // Marko's coauthors are those people who wrote the same books/articles/etc. as him marko.outE('wrote').inV.inE('wrote').outV.except([marko])[0].foreach{g.addEdge(null, x, it, 'hasCoauthor')} Conclusion Graphs are useful for modeling objects, their relationships to each other, and the conceptual structures wherein which they lie. From this explicit information, graph query and inference algorithms can be evaluated to answer questions on the graph and to increase the density of the explicit knowledge contained within the graph (i.e. increase the number of vertices and edges). This particular graph usage pattern has been exploited to a great extent in the world of RDF (knowledge representation) and RDFS/OWL (reasoning). The world of RDF/RDFS/OWL is primarily constrained to description logics (see an argument to the contrary here). Description logics are but one piece of the larger field of knowledge representation and reasoning. There are numerous logics that can be taken advantage of. In the emerging space of graph databases, the necessary building blocks exist to support the exploitation of other logics. Moreover, these logics, in some instances, may be used concurrently within the same graphical structure. To this point, the reading list below provides a collection of books that explicate different logics and ideas regarding heterogeneous reasoning. Graph databases provide a green field by which these ideas can be realized. Further ReadingBrachman, R., Levesque, H., “Knowledge Representation and Reasoning,” Morgan Kaufmann, 2004. Wang, P., “Rigid Flexibility: The Logic of Intelligence,” Springer, 2006. Mueller, E.T., “Commonsense Reasoning,” Morgan Kaufmann, 2006. Minsky, M., “The Society of Mind,” Simon & Schuster, 1988.Reference: Knowledge Representation and Reasoning with Graph Databases from our JCG partner Marko Rodriguez at the Marko A. Rodriguez’s blog blog....

Spring Security Misconfiguration

I recently saw Mike Wienser’s SpringOne2GX talk about Application Security Pitfalls. It is very informative and worth watching if you are using Spring’s stack on servlet container. It reminded me one serious Spring Security Misconfiguration I was facing once. Going to explain it on Spring’s Guide Project called Securing a Web Application. This project uses Spring Boot, Spring Integration and Spring MVC. Project uses these views:       @Configuration public class MvcConfig extends WebMvcConfigurerAdapter { @Override public void addViewControllers(ViewControllerRegistry registry) { registry.addViewController("/home").setViewName("home"); registry.addViewController("/").setViewName("home"); registry.addViewController("/hello").setViewName("hello"); registry.addViewController("/login").setViewName("login"); }} Where “/home”, “/” and “/login” URLs should be publicly accessible and “/hello” should be accessible only to authenticated user. Here is original Spring Security configuration from Guide: @Configuration @EnableWebMvcSecurity public class WebSecurityConfig extends WebSecurityConfigurerAdapter { @Override protected void configure(HttpSecurity http) throws Exception { http .authorizeRequests() .antMatchers("/", "/home").permitAll() .anyRequest().authenticated(); http .formLogin() .loginPage("/login") .permitAll() .and() .logout() .permitAll(); }@Override protected void configure(AuthenticationManagerBuilder auth) throws Exception { auth .inMemoryAuthentication() .withUser("user").password("password").roles("USER"); } } Nice and explanatory as all Spring’s Guides are. First “configure” method registers “/” and “home” as public and specifies that everything else should be authenticated. It also registers login URL. Second “configure” method specifies authentication method for role “USER”. Of course you don’t want to use it like this in production! Now I am going to slightly amend this code. @Override protected void configure(HttpSecurity http) throws Exception {         //!!! Don't use this example !!!         http             .authorizeRequests()                               .antMatchers("/hello").hasRole("USER");                  //... same as above ... } Everything is public and private endpoints have to be listed. You can see that my amended code have the same behavior as original. In fact it saved one line of code. But there is serious problem with this. What if my I need to introduce new private endpoint? Let’s say I am not aware of the fact that it needs to be registered in Spring Security configuration. My new endpoint would be public. Such misconfiguration is really hard to catch and can lead to unwanted exposure of URLs. So conclusion is: Always authenticate all endpoints by default.Reference: Spring Security Misconfiguration from our JCG partner Lubos Krnac at the Lubos Krnac Java blog blog....

Builder pattern using Java 8

I work in an environment where a great deal of our day to day scripting tasks occur through calling remote services as opposed to working with the database. For a lot of scripting tasks I’ve often used Groovy and one of the most useful features of Groovy specifically for that task has been it’s built in fluent Builders. Now Groovy builders exploit a few Groovy language features that are never going to make it into Java. Most notably Groovy builders make use of Groovy’s Meta programming features which isn’t coming to Java any time soon. However a key feature that Groovy builders have is their hierarchical approach to building constructs. This allows the builders to neatly and safely create nested tree like constructs which can be used to model everything from UX form layouts to XML. This approach we can at least model quite succinctly using Java 8 lambda expressions. For my sample I decided to take a reasonably simple Maven pom file and see if I could create a builder to handle that. All the code for the builder is available on Github here. The pom.xml file is as follows: <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.github</groupId> <artifactId>lambda-builder</artifactId> <version>1.0-SNAPSHOT</version> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.11</version> </dependency> <dependency> <groupId>commons-beanutils</groupId> <artifactId>commons-beanutils</artifactId> <version>1.7.0</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.8</source> <target>1.8</target> <fork>true</fork> <compilerArgument>-proc:none</compilerArgument> </configuration> </plugin> </plugins> </build> </project> Here is the sample code for the builder to build this model: MarkupBuilder pom = new XmlMarkupBuilder(true, "pom") .at("xmlns", "http://maven.apache.org/POM/4.0.0") .at("xmlns:xsi", "http://www.w3.org/2001/XMLSchema-instance") .at("xsi:schemaLocation", "http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"); pom.el("modelVersion", "4.0.0"); pom.el("groupId", "com.github"); pom.el("artifactId", "lambda-builder"); pom.el("version", "1.0-SNAPSHOT"); pom.el("dependencies", () -> { pom.el("dependency", () -> { pom.el("groupId", "junit"); pom.el("artifactId", "junit"); pom.elx("version", version::get); }); pom.el("dependency", () -> { pom.el("groupId", "commons-beanutils"); pom.el("artifactId", "commons-beanutils"); pom.elx("version", version::get); }); }); pom.el("build", () -> { pom.el("plugins", () -> { pom.el("plugin", () -> { pom.el("groupId", "org.apache.maven.plugins"); pom.el("artifactId", "maven-compiler-plugin"); pom.el("configuration", () -> { pom.el("source", 1.8); pom.el("target", 1.8); pom.el("fork", true); pom.el("compilerArgument", "-proc:none"); }); }); }); }); A few notes on this in general:I created a special form of some the methods which takes a java.util.function.Supplier as a parameter, and allows you to delay evaluation of a value until you traverse the builder. I eschewed method chaining (although I catered for it in the builder). Trying both methods I personally felt this was a lot cleaner. Java doesn’t have all syntax sugar that Groovy has, so I used a java.lang.Runnable for the functional interface which reduced the syntax creating a closure, with the downside that you have to have a handle on the initial builder object.Nowhere as nice as Groovy builders but nevertheless a great step forward. Can’t wait for Java 8.Reference: Builder pattern using Java 8 from our JCG partner Julian Exenberger at the Dot Neverland blog....

Introduction : How to save data in your Android application

This is the first post in a series explaining the various ways to save data and application state in an Android application. There are many mechanisms in the Android SDK that can be used to save data, and it is something confusing to decide which one to use and when the operation should be triggered. One of the first and easiest things you’ll want to do when building your application is keeping data entered by the user and explicitly saved. Here are the most commonly used persistent storage modes :        The most basic way to save data is to save it to a file on the external storage of the device. This type of storage is accessible by all the other applications and by the user. It can also be seen when the device is mounted as external storage. The files will be kept even if your application is uninstalled unless you get the folder for which to save the files using the GetExternalFileDir() method. This is the best for files that are edited by the user and should be kept even if your application is removed, for example pictures and documents. You can also save files to an internal storage that is private for your application. In that case, the files are not accessible by the user or by other applications and will be deleted when the application is uninstalled. This is best if you want to save simple data to a file while avoiding the overhead of managing a database. Also, you can save user preferences as key/value pairs with the Preference API to create a SharedPreferences file. You can create an activity to save preferences using the PreferenceActivity or an activity that hosts a PreferenceFragment, both containing Preferences object in their layout instead of view objects. Finally, if you have data with a complex structure to save, each application has its own private SQLLite database to persist data. This data will be destroyed if the application is uninstalled.Also, there are many cases where the current state of the application needs to be saved because an event was triggered. For example :When the device is rotated, the current activity is recreated so the layout can be adjusted. When your application is sent to the background by the user or by another application, for example if a phone call is received. Also, if another activity in your application is by the user, the current activity is also sent to the background. An activity that is sent to the background can be recycled if the device needs more memory and is recreated when needed again. When the application is explicitly closed by the user.In those cases, you may need to save the data the user entered in a permanent data store. When the activity is sent to the background or on its way to be destroyed, the onPause event is always raised, followed by the onStop event if the IU is completely hidden, so saving persistent data is best done during the onPause event. Later on, when the user accesses the activity again, the onRestart, onStart and onResume events will be raised if the activity was still in memory and is resumed and the onCreate, onStart, and onResume events will be raised if the activity was recycled and needs to be recreated. So, if you need to display the data again in all cases, you should restore the UI during the onResume. If it is not necessary to save the data permanently and you only want to save the state of the UI, you can use the onSaveInstanceState event to store the state in a Bundle. This should not be relied upon to save data since the event is not part of the activity lifecycle and is only triggered by the UI  when the activity needs to be recreated or is sent to the background, but not when it is destroyed permanently : it is meant for storing transient view states. Some of the data is already saved by the Android SDK, but you may need to save extra information, for example if you have custom controls. When the user navigates back to the activity and the state of the UI needs to be restored, the bundle containing the state information is accessible from the onRestoreInstanceState event raised if the activity was still in memory, or from the onCreate event raised if the activity was recycled and needs to be recreated.Reference: Introduction : How to save data in your Android application from our JCG partner Cindy Potvin at the Web, Mobile and Android Programming blog....

Cluster-wide Java/Scala application deployments with Docker, Chef and Amazon OpsWorks

Docker is great for running isolated containers on a single node. However, most software systems run on multiple nodes, so in addition to Docker, we need some way of specifying which containers should run on which nodes.The specific problem that I want to solve is the following: I have two Scala daemons, which I want to run on multiple nodes (depending on configuration, each node can run one of the daemons, or both). I want a reasonably fast way to deploy modified binaries across the cluster. I also don’t want to spend too much time setting up the servers. (My Gentoo days are long over.) The end solution to which I arrived involves Docker, OpsWorks, Chef and Vagrant. But, taking things step by step. By the way – how would you solve the above problem? Please comment. Packaging the Java/Scala application First I need to be able to package and upload the binaries. Here Docker is perfect. I wrote a simple Dockerfile which:is based on a trusted ubuntu+java7 image – no need to install java on the servers! copies the fat-jars from my disk to the image specifies the entrypoint to run Java with the copied jarsThe full Dockerfile is available here: https://gist.github.com/adamw/166b82ec04c9c0f67453. Having such an image, I can push it to the (public or private) Docker registry, where it is available for download by the nodes in the cluster. If needed, I could also install any other OS-level dependencies that my app would require, and not worry about version conflicts and setting them up on the actual servers. If you take a look at the Dockerfile, you may notice that there are two jars. This is done to minimise the size of the Docker image that has to be uploaded after each code change. The first jar contains only the dependencies (Scala library, logging libraries, frameworks etc.). The second jar contains compiled application code. When a Docker image is build from a Dockerfile, a series of intermediate images are created, one after each step. For same commands, involving same files, no new images are created, but images are re-used from the Docker cache. Dependencies change rarely, so usually the dep-jar is unchanged, and hence a cached version is re-used (and the intermediate image is uploaded once). On the other hand, the application code changes always. It is important that the dependencies jar is added to the image first, so that the intermediate image contains the deps, but not the app code (which changes). In the end, usually an upload of only 2-3MB is needed. There’s one thing to note here, though. When determining if an image can be re-used after an ADD command (which copies a file from the local disk to the image), Docker simply checks the last-modified timestamp of the file. This would cause the dependencies fat-jar to be re-added each time it is rebuilt, even if it’s identical. Hence I created a simple bash script, which copies the fat-jars next to the Dockerfile (from where they are uploaded as part of the Docker context) only if their md5 checksum changed: https://gist.github.com/adamw/ba5d8b79ff553fba83fd. How to create such two separate jars with SBT? Quite simple. Just use the SBT Assembly plugin and change its settings: assemblyOption in assembly ~= { _.copy(includeBin = true, includeScala = false, includeDependency = false) } The assemblyPackageDependency goal will then create the dependencies-only jar, and assembly will create the app-only jar. Setting up servers With the Docker image containing our app waiting in the cloud (on the Docker hub), it’s now time to setup the servers, where the Docker daemon will run the containers. For provisioning the servers I chose Chef with Amazon OpsWorks, for a couple of reasons: it is possible to clearly separate and organise the EC2 instances using Stacks and Layers, the servers come with an out-of-the-box integration with Chef, and it is very easy to use custom Chef recipes. Manual instance setup isn’t needed at all! The steps below are in part a summary, in part an extension of what’s described on ShopIgniter’s blog. The Chef setup (ran by OpsWorks) is going to be minimal, and only include what’s required to run Docker. First, we need to create an Ubuntu 12.04-based AMI with an updated kernel (14.04 doesn’t yet work with OpsWorks) – please refer to ShopIgniter’s blog for details. Secondly, we will be using custom Chef recipes; for that, you need to create a dedicated repository (e.g. on GitHub). The recipes are quite basic and simple: https://gist.github.com/adamw/792f8c22abb09699b6d5. Summing them up:docker::setup installs Docker docker::kill_containers kills & removes all running containers docker::myapp pulls the myapp image from the Docker registry and runs a container with command-line parameters and environmental variables as specified in a per-app section of the Chef-JSON configuration file, for example (here our app takes a single command-line parameter and requires AWS credentials in the environment):{ "myapp": { "image": "adamw/myapp:latest", "cmdline": [ "com.softwaremill.myapp.Main", "10" ], "env": { "AWS_ACCESS_KEY_ID": “...", "AWS_SECRET_ACCESS_KEY": “..." } } } Configuring OpsWorks To configure OpsWorks, we need to create a Stack, using a custom Chef cookbook, and a custom configuration JSON, for example as the one above (for each app / type of container we want to run, we need a section in the configuration JSON). Secondly, for each application (container) that we want to deploy, we need to create a Layer. As the Layers will only run Docker, we are not using any of the pre-configured Layers, and using a “Custom” one. The Layer will contain our custom recipes: in the Setup phase, we need to use the docker::setup recipe, and in the Deploy phase, we need to use the docker::kill_containers and docker::myapp recipes. Now, every time the Deploy phase is run on the layer, Docker will pull the images, and run the specified containers! By creating layers with appropriate recipes, we can start any combination of containers, on any node.Running the deploy phase To actually run the Deploy phase with a single click, we need to create a dummy OpsWorks App: just choose “Type: Other” and “Repository type: Other”. Now, each time you want to deploy the application on your servers (run updated Docker containers), just deploy this dummy app on the instances or layers you want. This can also be done with an API call (as everything on AWS)! So the whole process of building the app, creating the Docker image, pushing it, and running deploy on OpsWorks can be quite easily automated – for example after a successful build. With all pieces in place, we can now add new instances to the Layers, start and stop them, and have a multi-node cluster running our application! To update the app, just push the binaries to the registry. Testing Chef locally While the Chef recipes are quite minimal, it is still useful to be able to test them locally. That is easily possible with Vagrant. Using Vagrant, we can easily create a VM with Chef installed, which runs our recipes – and in result, the Docker containers. The Vagrantfile for this particular case is here: https://gist.github.com/adamw/bf6fa803b6b13fd7430b. The Vagrantfile contains a reference to the Chef recipes we are developing (through chef.cookbooks_path), and has the same configuration JSON as we use in OpsWorks. After issuing vagrant up, we will have the VM running. After changing the recipes, or uploading a new container, we can easily re-run the Chef recipes by using vagrant provision --provision-with chef_solo. Summing up We end up with the following separation of concerns:Docker – running applications in isolated containers, with all required dependencies Chef – setting up docker, running and linking containers with specified parameters/environment on defined nodes OpsWorks – managing instances, triggering deployment Vagrant – local testing of the whole setupWhile there are certainly things to simplify in the whole process described above (and I hope project Atomic will do just that!), in the end deploying a new version of your modified app across the cluster is easy and pain-free, providing a great development environment.Reference: Cluster-wide Java/Scala application deployments with Docker, Chef and Amazon OpsWorks from our JCG partner Adam Warski at the Blog of Adam Warski blog....

Git : How to add commit in between old commits

I have a Git repository and need to rewrite my local history by inserting a new commit in between old commits. More specifically my sitatution is like this:               AB—BC—CD—EF MASTER and I wanted to come up with something like this: AB—BC—SA—CD—EF MASTER Where SA is my new commit i.e to be inserted b/w commit BC & CD. Well, it isn’t actually an extreme or difficult case.It’s actually a very simple procedure to follow: $ git checkout master $ git checkout -b temp BC $ git add $ git commit # your changes that will be SA Now your Repo will look like this : AB—BC—SA temp \ CD—EF MASTER After this repository layout it’s rather simple to transform it into a single sequence of commits: $ git rebase temp master You may get few conflicts that you need to resolve . Now you are all Done !!! TAGGING COMMITS You will notice that your SHA keys are modified and tag doesn’t appear above commit BC to make sure your tags are in line follow the steps: $ git tag -l #to list all your tags. For each tag type the following command, $ git show TAG_NAME to see the details of the old commit. Make note of the subject line, date, and hash of the old commit. Page through git log looking for that subject line and date. Make note of the hash of the new commit when you find it. $ git tag --force TAG_NAME NEW_COMMIT_HASH #to update the tag. Hope that you have not mistaken one commit for another with a similar subject line and date. Thanks to Santosh Mohanty for writing this post .Post Comments And Suggestions !!!Reference: Git : How to add commit in between old commits from our JCG partner Abhishek Somani at the Java, J2EE, Server blog....

Apache Ant tasks for JMX access

I wanted to invoke JMX operations from the Ant tasks. However finding usable ant tasks library as well as the usage was rather tricky. So let me share my experience to make things easier for others. Ant tasks for JMX operations I decided to follow Tomcat documentation and used ant tasks distributed with tomcat. Just for the record the usage is not restricted to Tomcat deployed JMX mBeans. For me it worked for java process accessible via JConsole via Remote connection. Retrieving the libraryAs I wanted to get the latest version I used maven central repository “search by classname” feature and searched for: org.apache.catalina.ant.jmx.JMXAccessorTask (see the query) afterwards I went for the tomcat 8 jar file (called tomcat-catalina-ant-8.0.8.jar) and just copyied the latest available to my $ANT_HOME/lib dir.Usage I didn’t have a chance (or motivation?) to check all the tasks available, the full list of tasks available can be seen in the zipped file: org/apache/catalina/ant/jmx/antlib.xml, following were present for me: <typedef name="open" classname="org.apache.catalina.ant.jmx.JMXAccessorTask" /> <typedef name="set" classname="org.apache.catalina.ant.jmx.JMXAccessorSetTask" /> <typedef name="get" classname="org.apache.catalina.ant.jmx.JMXAccessorGetTask" /> <typedef name="invoke" classname="org.apache.catalina.ant.jmx.JMXAccessorInvokeTask" /> <typedef name="query" classname="org.apache.catalina.ant.jmx.JMXAccessorQueryTask" /> <typedef name="create" classname="org.apache.catalina.ant.jmx.JMXAccessorCreateTask" /> <typedef name="unregister" classname="org.apache.catalina.ant.jmx.JMXAccessorUnregisterTask" /> <typedef name="equals" classname="org.apache.catalina.ant.jmx.JMXAccessorEqualsCondition" /> <typedef name="condition" classname="org.apache.catalina.ant.jmx.JMXAccessorCondition" /> out of these, I gave following a try: org.apache.catalina.ant.jmx.JMXAccessorTask org.apache.catalina.ant.jmx.JMXAccessorInvokeTask org.apache.catalina.ant.jmx.JMXAccessorQueryTask For the demonstration purposes I’m using Glassfish 4.0. Example: Listing JMX MBeans Let’s assume we want to retrieve the MBean by name (namely: java.lang:type=Memory). Please note username and password were not required for access (otherwise they should be specified via respective properties). Noteworthy here is the resultproperty, which could hold array from which we could get a name. So having in ant build script: <typedef name="jmxQuery" classname="org.apache.catalina.ant.jmx.JMXAccessorQueryTask" /><jmxQuery host="localhost" port="8686" echo="true" name="java.lang:type=Memory" resultproperty="memory" /><echo>Retrieved MBeans count: ${memory.Length}</echo> <echo>The 1.st one has name: ${memory.0.Name}</echo> results for me in following output: [jmxQuery] memory.Length=1 [jmxQuery] memory.0.Name=java.lang:type=Memory [echo] Retrieved MBeans count: 1 [echo] The 1.st one has name: java.lang:type=Memory Example: Invoking operation via JMX Here is a 2 step approach required:connect to remote server via JMX and afterwards invoke the operation on the particular MBean.For demonstration purposes, let’s assume we want to call garbage collection (via invoking operation: gc() on MBean named: java.lang:type=Memory) Sample ant build file chunk does the job (please note ref property value that has to be the same across these 2 tasks): <typedef name="jmxOpen" classname="org.apache.catalina.ant.jmx.JMXAccessorTask" /><typedef name="jmxInvoke" classname="org.apache.catalina.ant.jmx.JMXAccessorInvokeTask" /><jmxOpen host="localhost" port="8686" ref="glassfish" failOnError="true" /><jmxInvoke name="java.lang:type=Memory" operation="gc" echo="true" delimiter=" " failOnError="true" ref="glassfish" /> Further information For more details, I recommend reading the official Tomcat documentation as well as Javadocs.Reference: Apache Ant tasks for JMX access from our JCG partner Peter Butkovic at the pb’s blog about life and IT blog....

Testing with Aliens: How to test a JPA type converter with Arquillian

This post was written together with +Aslak Knutsen (@aslakknutsen). JPA type converters provide an easy way to define how an entity attribute gets persisted to the database. You can use them to implement lots of different features, e.g. to encrypt your data as I showed in a previous post: How to use a JPA Type Converter to encrypt your data. But writing the type converter is not enough. We also need to make sure, that it is working correctly. In general, there are two ways to test a type converter. We could write a unit test to check, if the conversion works correctly. But a unit test performs a test of the isolated class without putting it into the real execution environment. That means that we will still not know, if the converter works in one of our applications. If everything is set up correctly, the persistence provider will call the converter before writing to and after reading from the database. So we also need to check if the type converter gets called by the persistence provider and if everything works fine under that condition. We need to test the converter inside of the container we want to use for our application. We will have a look at how this can be done with Arquillian and its persistence extension. Something about Arqillian If you are already familiar with Arquillian, you can skip this part. For all of you who have never worked with Arquillian so far, I just want to give some basic information. You can find a more detailed description at the Arquillian Getting Started guide. Arquillian is a test framework for in container testing. The idea is to do not mock the container you want to use but to test your code inside of it. This provides the advantage, that you can test if your code will also work in your execution environment and not only in your mocked up test scenario. Arquillian provides lots of functionality to manage the container, inject required resources like EJBs or an EntityManager and make your live much easier. The Arquillian tests are executed by junit. This is great, because you can use them everywhere, where you can execute junit tests. And that means in your IDE, as part of your build process, on your CI server, simply everywhere. Object under test The following code snippet shows the object under test for this example. It is a type converter that encrypts and decrypts a String attribute. The converter gets called by the persistence provider before writing to and after reading from the database. If you want to read more about how this type converter works, check my posting about it. @Converter public class CryptoConverter implements AttributeConverter<String, String> {private static final String ALGORITHM = "AES/ECB/PKCS5Padding"; private static final byte[] KEY = "MySuperSecretKey".getBytes();@Override public String convertToDatabaseColumn(String ccNumber) { // do some encryption Key key = new SecretKeySpec(KEY, "AES"); try { Cipher c = Cipher.getInstance(ALGORITHM); c.init(Cipher.ENCRYPT_MODE, key); return Base64.encodeBytes(c.doFinal(ccNumber.getBytes())); } catch (Exception e) { throw new RuntimeException(e); } }@Override public String convertToEntityAttribute(String dbData) { // do some decryption Key key = new SecretKeySpec(KEY, "AES"); try { Cipher c = Cipher.getInstance(ALGORITHM); c.init(Cipher.DECRYPT_MODE, key); return new String(c.doFinal(Base64.decode(dbData))); } catch (Exception e) { throw new RuntimeException(e); } } } Setting it up Before we can start to write our tests, we need to define a few dependencies. I will only show how to configure the dependencies we need for this example. If you have not already set up arquillian tests for your project, you will have to do a little bit more. Please check the Getting Started guide to learn how to setup arquillian for your project. Don’t be afraid, there is not too much to do. As you can see in the following snippet, we will use JUnit 4.11, Arquillian 1.1.3.Final, the Arquillian Persistence Extension 1.0.0.Alpha7 and the WildFly Application Server 8.1.0.Final. <?xml version="1.0" encoding="UTF-8"?> <project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> ...<properties> <version.junit>4.11</version.junit> <version.arquillian>1.1.3.Final</version.arquillian> <version.arquillian_persistence>1.0.0.Alpha7</version.arquillian_persistence> <version.wildfly>8.1.0.Final</version.wildfly> </properties><dependencyManagement> <dependencies> ... <dependency> <groupId>org.jboss.arquillian</groupId> <artifactId>arquillian-bom</artifactId> <version>${version.arquillian}</version> <type>pom</type> <scope>import</scope> </dependency> </dependencies> </dependencyManagement><dependencies> ... <dependency> <groupId>javax.enterprise</groupId> <artifactId>cdi-api</artifactId> <scope>provided</scope> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>${version.junit}</version> <scope>test</scope> </dependency> <dependency> <groupId>org.jboss.arquillian.junit</groupId> <artifactId>arquillian-junit-container</artifactId> <scope>test</scope> </dependency> <dependency> <groupId>org.jboss.arquillian.extension</groupId> <artifactId>arquillian-persistence-dbunit</artifactId> <version>${version.arquillian_persistence}</version> <scope>test</scope> </dependency> </dependencies> Writing the tests There are two things we need to do to setup our test environment. At first, we need to tell junit that this test shall be executed as a junit test. This is done by @RunWith(Arquillian.class). Additionally, we need to create the test deployment, that will be deployed to the container. Therefore we need to implement at least one method and annotate it with @Deployment. As you can see in the following code snippet, we use ShrinkWrap to create a jar archive deployment. The archive contains the CreditCard entity, the CryptoConverter type converter and the test class. There is no need to include any EJBs or other classes which implement business logic. We can inject the EntityManager into our test case and use it directly to persist and read entities. We will have a more detailed look at it later on. Additionally, we need to add some manifest resources to create a persistence unit, register the type converter and add an empty beans.xml to activate CDI. Please check the getting started guide to get more information on ShrinkWrap and creating deployments. @RunWith(Arquillian.class) public class TestCryptoConverter {@Deployment public static JavaArchive createDeployment() { return ShrinkWrap .create(JavaArchive.class) .addClasses(CreditCard.class, CryptoConverter.class, TestCryptoConverter.class) .addAsManifestResource("META-INF/persistence.xml", "persistence.xml") .addAsManifestResource("META-INF/orm.xml", "orm.xml") .addAsManifestResource(EmptyAsset.INSTANCE, "beans.xml"); } After this is done, we can start to write the test cases. At first, we will persist a CreditCard entity and check, if the credit card number gets encrypted by the CryptoConverter. Therefore we inject the EntityManager, create a CreditCard entity and pass it to the persist method of the EntityManager. The validation of the persisted data, is done by the Arquillian persistence extension. We just need to define the data we expect to be written to the database. The expected data is defined in the cc.yml file, which is referenced in the @ShouldMatchDataSet annotation. Because the id attribute is generated by the database, we want to exclude it from the validation. This can be done by referencing it in the excludeColumns attribute of the annotation. @PersistenceContext private EntityManager em;@Test @ShouldMatchDataSet(value = "data/cc.yml", excludeColumns = "id") public void testEncryption() { CreditCard cc = new CreditCard(); cc.setName("My Name"); cc.setCcNumber("123456789");this.em.persist(cc); } The cc.yml contains the following information. CreditCard: - id: 1 name: My Name ccNumber: egFfkhd8cRh82tvsh3VVUg== In the second test, we will check if we can search the database for a CreditCard entity with a given credit card number. Therefore we use the @UsingDataSet annotation to seed the database with data defined in the cc.yml file. Now we can use the injected EntityManager to call a named query to search for CreditCard entity with the given number. @Test @UsingDataSet("data/cc.yml") public void testRead() { CreditCard cc = this.em .createNamedQuery(CreditCard.BY_NUMBER, CreditCard.class) .setParameter("number", "123456789").getSingleResult(); Assert.assertEquals("My Name", cc.getName()); } Conclusion We used Arquillian and the Arquillian Persistence extension to test a JPA type converter. Therefore we injected the EntityManager and used the annotations @ShouldMatchData and @UsingDataSet to validate and seed the database with a yml file. If you want to try it yourself, you can find the sources on github. You can run it by calling: git clone https://github.com/thjanssen/JPA2.1.git && cd JPA2.1/CryptoConverter && mvn test. What are your experiences with testing your Java EE application with Arquillian? Please write a comment about it. Want to learn more about Arquillian, see the Arquillian Guides: http://arquillian.org/guides/Reference: Testing with Aliens: How to test a JPA type converter with Arquillian from our JCG partner Thorben Janssen at the Some thoughts on Java (EE) blog....

JavaMail can be evil (and force you to restart your app server)

JavaMail always had an interesting approach when it comes to its configuration. Basically you have to fill an untyped map or Properties structure and hope for the correct interpretation. Countless tutorials on the net show the minimal properties required to make it work (send / receive mails). However, as we painfully just learned, there are some lesser known properties you should probably take care of, which is timeout settings for socket IO. By default, JavaMail uses an infinite timeout for all socket operations (connect, IO, …)! Now suppose you have a cluster of SMTP servers which handle outgoing mail, accessed via a DNS round robin. If one of those servers fail, which happens to be the one JavaMail wanted to connect to, your mail sending thread will hang – forever! This is exactly what happened to us and we needed to perform some real nasty magic to avoid tragedy. Therefore, we now set timeouts for all operations: String MAIL_SMTP_CONNECTIONTIMEOUT ="mail.smtp.connectiontimeout"; String MAIL_SMTP_TIMEOUT = "mail.smtp.timeout"; String MAIL_SMTP_WRITETIMEOUT = "mail.smtp.writetimeout"; String MAIL_SOCKET_TIMEOUT = "60000";// Set a fixed timeout of 60s for all operations - // the default timeout is "infinite" props.put(MAIL_SMTP_CONNECTIONTIMEOUT, MAIL_SOCKET_TIMEOUT); props.put(MAIL_SMTP_TIMEOUT, MAIL_SOCKET_TIMEOUT); props.put(MAIL_SMTP_WRITETIMEOUT, MAIL_SOCKET_TIMEOUT); Also, if you plan to access DNS round robin based services (like amazon S3) or in our case a mail cluster, don’t forget to also configure the DNS cache tiemout of Java (which is also infinite by default): // Only cache DNS lookups for 10 seconds java.security.Security.setProperty("networkaddress.cache.ttl","10"); And while we’re at it, for us it turned out to be a good idea to set all encodings to UTF-8 (independent of the underlying OS) to provide a stable environment: System.setProperty("file.encoding", Charsets.UTF_8.name()); System.setProperty("mail.mime.charset", Charsets.UTF_8.name());…you don’t want to care about stuff like this at all? Feel free to use our open source Java library SIRIUS, which takes care of all that by providing a neat fluet API for sending mails: Sources on GitHub. An example usage can be found in the cluster manager: @Part private MailService ms;private void alertClusterFailure() { ... ms.createEmail() .useMailTemplate("system-alert", ctx) .toEmail(receiver).send(); ... }Reference: JavaMail can be evil (and force you to restart your app server) from our JCG partner Andreas Haufler at the Andy’s Software Engineering Corner blog....

10 things you can do to make your app secure: #3 Validate Input

This is part #3 of a series of posts on the OWASP Top 10 Proactive Development Controls. Your first line of defence against attacks should always be to check all data from untrusted sources. Input validation is fundamental to application security, and a basic part of good defensive programming. This is simple, and obvious – and often done wrong.       Don’t Rely on Client-Side Validation One common mistake is relying on client-side validation to catch problems. Client-side validation is useful in providing immediate feedback at the UI. But it won’t protect your system, especially in web apps. If an attacker can find a way to insert themselves in between the browser and your app, which can be done using a proxy, they can play with any of the data, including header fields and other hidden data, after local client-side editing has already been done. Data from a client, especially a client outside of your network, should never be trusted. Another common mistake is relying on negative checking or “black list” validation to try to catch known bad data. This is a weak technique: you can’t be sure that you will catch all of the bad things or combinations, and negative checks can be subverted through (double) encoding and other evasion tricks. White List Validation OWASP recommends that you always use positive restrictions – white list validation – where you define the acceptable type and size, and all allowable values using regular expressions (regex). An example from the Proactive Controls: the regex for a password field could be:^(?=.*[a-z])(?=.*[A-Z]) (?=.*\d) (?=.*[@#$%]).{10,64}$This regular expression ensures that a password is 10 to 64 characters in length and includes a uppercase letter, a lowercase letter, a number and a special character (one or more uses of @, #, $, or %). White list validation is trivial for enumerated data: days of the week, months of the year … anything that you can fit into a defined list. For other common data types (such as dates and time zones, numeric types, currencies, email addresses, IP addresses, URLs and domain names, and credit card numbers), you could use the Apache Commons Validator library. Validation of free-format text fields (names, comments) is more difficult, and especially text fields that can – or need to – contain special characters, especially markup:If your application handles markup — untrusted input that is supposed to contain HTML — it can be very difficult to validate. Encoding is also difficult, since it would break all the tags that are supposed to be in the input. Therefore, you need a library that can parse and clean HTML formatted text such as the OWASP Java HTML Sanitizer. A regular expression is not the right tool to parse and sanitize untrusted HTML. You can test how solid your input validation is with different tools, such as fuzzers or static analysis taint checking (using tools that run through execution paths in the code and identify when you are referencing data that has not been validated) and through pen testing and manual exploratory testing.Reference: 10 things you can do to make your app secure: #3 Validate Input from our JCG partner Jim Bird at the Building Real Software blog....
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below: