Featured FREE Whitepapers

What's New Here?

apache-maven-logo

When Maven Dependency Plugin Lies

Problem: We had an integration test which creates a spring ClassPathXmlApplicationContext and while doing so the test blew up with a NoSuchMethodError. It turned out that we had conflicting versions of dependencies on Spring artifacts. This in itself is not an unusual problem – such problems are addressed with the maven dependency plugin using the verbose option. However, what do you do when the maven plugin is wrong?       Investigation: We started to dig in and we found that the AbstractAutowireCapableBeanFactory in its getTypeForFactoryMethod method tries to access the GenericTypeResolver resolveReturnTypeForGeneric method and fails on a java.lang.NoSuchMethodError: org.springframework.core.GenericTypeResolver.resolveReturnTypeForGenericMethod(Ljava/lang/reflect/Method;. Initial investigation and googling we found that the method was added in 3.2.0 while we’re supposed to be running with 3.1.1. Further investigation determined that spring-data-mongodb depends on spring framework in range [3.0.7-4) 1 and because maven takes the latest available version given that range 2 it tried to take 3.2.2. Note that the above changes a bit given a conflict between an explicit version dependency and a range dependency but, IINM, when determining the dependencies of spring mongo there is no conflict. The problem was further masked by two symptoms:We have other projects that use this pattern and have no problem- This was explained by the fact that the conflict-resolving mechanism of maven chooses the nearest version it finds by default 3 and since all other projects which need spring-data-mongodb depend on this project they were lucky enough to grab the 3.1.1 version and not 3.2.2 dependency:tree shows it brings 3.1.1 while bringing 3.2.2- Since the stack trace showed other results I wrote a test which checks from which jar each of the above classes comes from and verified that indeed the AbstractAutowireCapableBeanFactory class arrives from spring-beans 3.2.2 and not 3.1.1 as “mvn dependency:tree” showed (a big thanks to http://bit.ly/10zD1iV for the code snippet of finding the jar of a class in runtime).Maven dependency:tree output showing spring-beans:3.1.1 is used in the artifact &gt:mvn dependency:tree -Dverbose -Dincludes=org.springframework ... (omitted for clarity) ... [INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ wix-feature-toggle-administration --- [INFO] artifact org.springframework:spring-beans: checking for updates from central [INFO] artifact org.springframework:spring-beans: checking for updates from snapshots [INFO] artifact org.springframework:spring-expression: checking for updates from central [INFO] artifact org.springframework:spring-expression: checking for updates from snapshots [INFO] artifact org.springframework:spring-tx: checking for updates from central [INFO] artifact org.springframework:spring-tx: checking for updates from snapshots [INFO] com.wixpress.common:wix-feature-toggle-administration:jar:2.180.0-SNAPSHOT ... [INFO] +- org.springframework.data:spring-data-mongodb:jar:1.0.1.RELEASE:compile [INFO] | +- org.springframework:spring-beans:jar:3.1.1.RELEASE:compile [INFO] | | \- (org.springframework:spring-core:jar:3.2.2.RELEASE:compile - omitted for conflict with 3.1.1.RELEASE) [INFO] | +- org.springframework:spring-expression:jar:3.1.1.RELEASE:compile [INFO] | | \- (org.springframework:spring-core:jar:3.2.2.RELEASE:compile - omitted for conflict with 3.1.1.RELEASE) [INFO] | \- org.springframework.data:spring-data-commons-core:jar:1.2.1.RELEASE:compile [INFO] | +- (org.springframework:spring-beans:jar:3.1.1.RELEASE:compile - omitted for duplicate) [INFO] | \- (org.springframework:spring-tx:jar:3.1.1.RELEASE:compile - omitted for duplicate) [INFO] +- com.wixpress.common:wix-framework:jar:2.180.0-SNAPSHOT:compile [INFO] | +- org.springframework:spring-core:jar:3.1.1.RELEASE:compile [INFO] | | \- org.springframework:spring-asm:jar:3.1.1.RELEASE:compile ... I've removed additional outputs for clarity. The additional outputs were all 3.1.1 and were further down the tree (so irrelevant due to maven conflict resolving mechanism) Test which proves spring-beans:3.2.2 is used in the artifact (asserting what the jvm in the error was saying) package com.wixpress.springVersionBug;import org.junit.*; import org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory; import org.springframework.core.GenericTypeResolver; import java.security.CodeSource; import static org.hamcrest.Matchers.endsWith;/** * @author ittaiz * @since 3/24/13 */public class SpringVersionTest {@Test public void verifySpringBeans311InClasspath(){ verifyCorrectSpringVersionInClasspathFor(AbstractAutowireCapableBeanFactory.class,"spring-beans-3.1.1.RELEASE.jar"); } @Test public void verifySpringCore311InClasspath(){ verifyCorrectSpringVersionInClasspathFor(GenericTypeResolver.class,"spring-core-3.1.1.RELEASE.jar"); } public void verifyCorrectSpringVersionInClasspathFor(Class springClass,String expectedJarFileName){ CodeSource springClassCodeSource = springClass.getProtectionDomain().getCodeSource(); Assert.assertNotNull("expecting "+expectedJarFileName+" to be loaded by non-system class loader",springClassCodeSource); Assert.assertThat(springClassCodeSource.getLocation().toString(),endsWith(expectedJarFileName)); } }The reason spring-core artifact came in 3.1.1 when spring-beans came as 3.2.2 is that our framework explicitly depends on spring-core and this artifact explicitly depends on the framework. This means spring-core 3.1.1 from framework is 2 hops which is shorter than the 3.2.2 from spring-data-mongodb. Solution: Depend on spring-data-mongodb while excluding spring-beans like so: <dependency> <groupId>org.springframework.data</groupId> <artifactId>spring-data-mongodb</artifactId> <version>1.0.1.RELEASE</version> <exclusions> <exclusion> <groupId>org.springframework</groupId> <artifactId>spring-beans</artifactId> </exclusion> </exclusions> </dependency> The Open question mark: Why dependency:tree (in verbose mode) did not show that it brings spring-beans in 3.2.2 but in 3.1.1 while explicitly specifying that spring-core 3.2.2 was removed due to conflict? I chalk this up to a bug in the dependency plugin.http://repo1.maven.org/maven2/org/springframework/data/spring-data-mongodb-parent/1.0.1.RELEASE/spring-data-mongodb-parent-1.0.1.RELEASE.pom ↩ http://www.maestrodev.com/better-builds-with-maven/creating-applications-with-maven/resolving-dependency-conflicts-and-using-version-ranges/ ↩ http://www.maestrodev.com/better-builds-with-maven/creating-applications-with-maven/resolving-dependency-conflicts-and-using-version-ranges/ ↩  Reference: When Maven Dependency Plugin Lies from our JCG partner Yoav Abrahami at the Wix IO blog. ...
devops-logo

Concepts of SNMP (including v3)

SNMPPurpose. SNMP is a protocol for getting the status (e.g., CPU load, free memory, network load) of computing devices such as routers, switches and even servers. Object descriptor, managed object. The client can provide a globally unique names such as cpmCPUTotal5secRev (the average CPU load of a Cisco device for the past 5 seconds) to indicate the information that it wants, then the server should return such information. Such a textual name is called the “object descriptor”. The word “object” or “managed object” refers to the concept of CPU load. The actual CPU load in the device is called the “object instance”.   Object identifier (OID). To make sure that each object descriptor is unique, actually it is defined using a list of integers such as 1.3.6.1.4.1.9.9.109.1.1.1.1.6. Each integer is like a package in Java. For example, the integers in 1.3.6.1.4.1.9 represents iso (1), org (3), dod, i.e., department of defense (6), internet (1), private (4), enterprises (1), cisco (9) respectively. This allows the Internet authority to delegate the management of the namespace hierarchically: to private enterprises and then to Cisco, which can further delegate to its various divisions or product categories. Such a list of integers is called an “object identifier”. This is the ultimate identification for the managed object.Even though the object descriptor should be unique, it is useful to see the hierarchy. Therefore, usually the full list of object descriptors is displayed such as iso.org.dod.internet.private.enterprises.cisco…cpmCPUTotal5secRev. Why use integers instead of symbolic names? Probably to allow the network devices (with little RAM or CPU power) implementing SNMP to save space in processing. Symbolic names such as object descriptor can be used by human in commands, but in the protocol’s operation it is done using object identifier. In principle, the object a.b.c.d and the object a.b.c.d.e on a device have NO containment relationship. That is, they are NOT like a Java object containing a child object. In fact, the value of each object in SNMP is basically a simple value (scalar) such as an integer or a string. The only relationship between them is their names.Identifying an instance. Now comes the most complicated concept in SNMP. Consider the concept of the number of bytes that have been received by a network interface on a router. This concept is an object. As a router should have multiple interfaces, there must be multiple instances of that object. Then, how can an SNMP client indicate to the SNMP server which instance it is interested in? The solution is more or less a kludge: to allow the instance of, say, a.b.c.d, to represent a table (a compound, structural value), which contains rows (also compound, structural value) represented by a.b.c.d.e. Each row contains child object instances (with scalar values only). Each child object is called a “columnar object”. For example, each row may contain three object instances: a.b.c.d.e.f, a.b.c.d.e.g, and a.b.c.d.e.idx. If you’d like to refer to the a.b.c.d.e.f instance in a particular row, you will write a.b.c.d.e.f.<index>. The meaning of the index is defined by a.b.c.d.e (the row). For example, it may be defined as finding the row in the table which contains a columnar object a.b.c.d.e.idx whose value equals to <index>, then return the columnar object a.b.c.d.e.f as the result.Note that this is the only situation where the value of an object can be a structure and that there is object containment relationship in SNMP. What is confusing is that a.b.c.d.e.f is used both as an object identifier and the lookup key to find the child instance in the row. Unlike other object identifiers, the identifier now represents an object containment relationship so it must have a.b.c.d.e as the prefix, otherwise the server won’t know which table to look into and what is the definition for the index. The complete identifier a.b.c.d.e.f.<index> is called an instance identifier. Here is a concrete example: Consider iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets.1.  The definition of iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry says that to find the row in the table, it should search for a row which contains a child object iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifIndex with the value of 1 (the index specified), then it will return the value of child object iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets in the row. Of course, for this to work, the  iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifIndex child object in each row must have been assigned with sequential values 1, 2, …, etc. (which is indeed the case).Finally, a simple case is that there is no table at all. For example, to find the up time of the device, use the object identifier iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemUptime and append a .0 as the index, so the instance identifier is iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemUptime.0. BTW, “hr” stands for “host resources”.MIB (management information base). A MIB is just a a collection of managed objects (not instances). There are standard MIBs so that device manufacturers can implement and users can find the right object identifiers to use. There are also proprietary MIBs such as those designed by Cisco to provide information only available on its devices. Finding the supported OIDs. How can you find out the MIBs or the object identifiers supported by a device? It is easier to just “walk the MIB tree”: to show all the instances in the tree or in a subtree. On Linux, this is done as below. You specify the IP or hostname of the server and optionally specify a node so that only that subtree is displayed (the object identifier starts with a dot, otherwise it will be assumed it is relative to iso.org.dod.internet.mgmt.mib-2):# snmpwalk <some args> localhost # snmpwalk <some args> localhost .iso.org.dod.internet.mgmt.mib-2.system # snmpwalk <some args> localhost systemGetting an instance. Just specify the instance identifier:# snmpget <some args> localhost .iso.org.dod.internet.mgmt.mib-2.host.hrSystem.hrSystemUptime.0 # snmpget <some args> localhost host.hrSystem.hrSystemUptime.0SNMP enttiy, engine and applications. SNMP is a peer to peer protocol. There is no concept of a server and a client (these terms are used here for simplicity). Instead, both peers have the same core capabilities such as sending or receiving SNMP messages, performing security processing (see below), integrating v1, v2c and v3 processing (dispatching) and etc. This part is called the SNMP engine. On top of the engine, there are different “applications”: one application may only respond to requests for object instances (called SNMP agent in v1 and v2), another application may probe others (called SNMP manager in v1 and v2), yet another may forward SNMP messages (SNMP proxy). The whole server or client is called the “SNMP entity”. Context. On some devices there are multiple copies of a complete MIB subtree. For example, a physical router may support the concept of virtual routers. Each such virtual router will have a complete MIB subtree of instances. In that case, each virtual router may be indicated as a “context” in the SNMP server. A context is identified by name. For example, a virtual router could be identified as the “vr1″ context. There is a default context with empty string (“”) as its name. When a client sends a query, it can specify a context name. If not, it will query the default context. Notification (trap). An SNMP server may actively send a notification to the client when some condition occurs. This is very much like a response without a request. Otherwise, everything is similar. The condition (e..g., the changes of the value of an instance or its going out of a range), the destination, the credentials used (see the security section below) and etc. are configured on the server. Transport binding. Typically SNMP runs on UDP port 161.SNMP securitySNMP v1 and v2c security. In SNMP v1 and v2c (v2 was not widely adopted), there is little security. The only security is the “community string”. That is, the server is configured to be in a community identify by a string such as “foo”, “public” (commonly used and the default for many devices to mean no protection) or “private”. If the client can quote the community strnig, then it is allowed access. As the community string is included as plain text in SNMP packets, it practically provides no security. Therefore, in v1 and v2c, to access an SNMP server, you will do something like:# snmpwalk -v 2c -c public localhost # snmpget -v 2c -c public localhost <INSTANCE ID>SNMP v3 security. In SNMP v3, there is user-based security. That is, the client may be required to authenticate the messages to the server as originating from a user using a password (authentication password). In addition, the client may be furthered required to encrypt the messages using another password (privacy password). This security requirement is called the “security level” (no authentication needed, authentication but no privacy, authentication with privacy). Therefore, in v3, you will access the server like:# snmpwalk -v 3 -l noAuthNoPriv localhost # snmpwalk -v 3 -l authNoPriv -u kent -A "my auth passwd" localhost # snmpwalk -v 3 -l authPriv -u kent -A "my auth passwd" -X "my priv passwd" localhostClient configuration file. To save typing all those every time, you can store these parameters into the snmp.conf file as defaults. Security limitation. It is a bad idea to specify the password on the command line as it can be revealed by local users using “ps”. Storing it into the configuration file is better. However, the file only allows a single authentication password and a single privacy password, not enough to handle the case of using different passwords for different servers.Security name. A security name is just a user name. No more, no less. That’s the term used in the RFC (maybe in the future it could be something else?) Algorithm. Further, there are different algorithms for authentication (HMAC using MD5 or SHA) and for privacy (encryption using DES or AES). So, you need to specify the algorithms to use:# snmpwalk -v 3 -l noAuthNoPriv localhost # snmpwalk -v 3 -l authNoPriv -u kent -a MD5 -A "my auth passwd" localhost # snmpwalk -v 3 -l authPriv -u kent -a MD5 -A "my auth passwd" -x DES -X "my priv passwd" localhostEnsure the algorithms match. As SNMP uses UDP and each query and response may use just a single UDP packet, there is no negotiation of algorithm at “connection phase” at all. In fact, presumably for simplicity in implementation, the algorithms used are not even indicated in the message, so the client must use the agreed-on algorithms as configured in the user account on the server, otherwise the server will simply fail to authenticate or decrypt the message.Localized keys. The authentication password and privacy password of a user account are not used directly. The idea is, most likely you will use the same password for all the user account on all devices on site. If it is directly used, then a hacker controlling one device will be able to find the password and use it to access all the other devices. Therefore, when creating a user account, you specify the password, but the Linux SNMP server will combine it with a unique ID (called the “engine ID”) generated for the device (such as the MAC or IP and/or a random number generated and stored on installation), hash it and use the result as the password (the “localized key”). This way, even if the hacker can find this localized key, he will still be unable to find the original password.But how can a client generate the same key? It has to retrieve the engine ID first and then perform the same hashing. This is supported by the SNMP protocol.User accounts creation. Due to the need to generate localized keys, the way to create user accounts on Linux is quite weird. You stop the server, specify the user account’s name and password in a file, then start the server. It will read the password, convert it to a localized key and overwrite the file. This file is /var/lib/snmp/snmpd.conf on Linux:createUser kent MD5 "my auth password" DES "my privacy password" createUser paul SHA "my auth password, no encryption needed"Access control. The access control can specify the user account, the lowest security level required, which part of the MIB tree is accessed (may use an OID to identify a subtree), the type of access (read or write), in order to grant the access. Here are some example settings on Linux (although human user names are used, but in practice they should be representing devices):rouser john noauth .iso.org.dod.internet.mgmt.mib-2.system rouser kent priv .iso.org.dod.internet.mgmt.mib-2.system rouser kent auth .iso.org.dod.internet.mgmt.mib-2 rwuser paul privView. How to specify several subtrees in an access control rule? You can define a view. A view has a name and is defined as including some subtrees and excluding some subtree. Then you can refer to it by name in access control:view myview included .iso.org.dod.internet.mgmt.mib-2.system view myview included .iso.org.dod.internet.mgmt.mib-2.host view myview excluded .iso.org.dod.internet.mgmt.mib-2.host.hrStoragerwuser paul priv -V myviewAccess control for v1 and v2c. For v1 and v2c, access control can specify the community string, the IP range of the client (the “source”), the subtree (OID) or the view:rocommunity public 192.168.1.0/24 .iso.org.dod.internet.mgmt.mib-2.system rwcommunity private localhost -V myviewMost flexible access control model. The above access control model is called the “traditional model”. The new, most flexible access control model is called the “view-based access control model (VACM)”, even though the former can also use views. It may be more suitable to called it group-based access control as it uses user groups in the rules (NOT the precise syntax yet!):group g1 kent group g1 paul #access <group> <context> <min sec level> <exact context?> <view for read> <view for write> <view for notify> access g1 "" auth exact myview1 myview2 myview3Mapping community string to user name. When using the VACM, instead of granting access to community strings, you need to merge v1 and v2c into the user-based access control processing. To do that, a community string along with the source can be mapped to a user name (the user name mapped to do NOT have to be existing):com2sec user1 192.168.1.0/24 public # "default" source means any com2sec user2 default privateSecurity model. Even though the different types of identity in the different SNMP versions are represented uniformly as a user name, their trustworthiness is still significantly different. So, in specifying group memberships and access control rules, you are required to specify the “security model” (v1, v2c or the user security model as in v3) and that’s the correct syntax:group g1 usm kent group g1 usm paul group g2 v2c user1 group g2 v1 user1 group g2 v1 user2 access g1 "" usm auth exact myview1 myview2 myview3 access g2 "" any noauth exact myview4 myview5 myview6   Reference: Concepts of SNMP (including v3) from our JCG partner Kent Tong at the Kent Tong’s personal thoughts on information technology blog. ...
java-logo

Why a synchronized StringBuffer was never a good idea

Introduction StringBuffer is a synchronized class for mutable strings.  The main problem with making it synchronized is thatIt was usually used as a local variable so making it synchronized just made it slower. It was never a good idea to use it in a multi-threaded way.  This problem is that developers assumed that methods which used StringBuffer were themselves thread safe when they were not.  The problem with StringBuffer This is an example from a real class which is used in production in many trading systems. It’s not a commonly used but you might assume that StringBuffer gives you thread safety, when it doesn’t. public void addProperty(String name, String value) { if (value != null && value.length() > 0) { if (sb.length() > 0) { sb.append(','); } sb.append(name).append('=').append(value); } } While individual calls are thread safe, multiple calls are not.  It is almost impossible to find a good use for StringBuffer that doesn’t involve multiple calls (including toString) A puzzle Imagine three threads call (in no particular order) T1: addProperty("a", "b"); T2: addProperty("c", "d"); T3: sb.toString(); write a program which will generate every possible output of T3’s sb.toString() I found 89.  With thread safety, you might reduce this to 4. Note If you used StringBuilder it would be worse, but at least you might not assume your method is thread safe when it is not. e.g. SimpleDateFormat uses StringBuffer  Reference: Why a synchronized StringBuffer was never a good idea from our JCG partner Peter Lawrey at the Vanilla Java blog. ...
java-logo

Simulation of time consuming actions in integration tests

Quite recently in one of my projects I had a situation in which I needed to create an integration test for the application. That’s not very odd isn’t it? What was interesting was the fact that the logic of the app involved some concurrency issues and one of the components had to connect to an external service which would take a couple of seconds. Since in the integration test there was no need to make the actual connection, the component needed to be mocked. What about the simulation of the time consuming action? Well, let’s take a look at the way I did it…       The task. package pl.grzejszczak.marcin;import org.slf4j.Logger; import org.slf4j.LoggerFactory;/** * Service that does some things including processing of the external service * * @author marcin * */ public class SomeTask implements Runnable {private static final Logger LOGGER = LoggerFactory.getLogger(SomeTask.class);// Service is injected via a dependency injection system private Processable timeConsumingExternalService;private void methodThatConnectsToExternalServices() { // connects to an external service and spends a couple of seconds there LOGGER.debug("Before processing"); timeConsumingExternalService.process(); LOGGER.debug("After processing"); // some other things to do }public void run() { methodThatConnectsToExternalServices(); }public void setTimeConsumingExternalService(Processable timeConsumingExternalService) { this.timeConsumingExternalService = timeConsumingExternalService; }} The integration test. package pl.grzejszczak.marcin;import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors;import org.mockito.Mockito; import org.mockito.invocation.InvocationOnMock; import org.mockito.stubbing.Answer; import org.slf4j.Logger; import org.slf4j.LoggerFactory;public class ServiceIntegrationTest {private static final Logger LOGGER = LoggerFactory.getLogger(ServiceIntegrationTest.class);private ExecutorService executorService = Executors.newCachedThreadPool(); private Processable timeConsumingExternalServiceMock = Mockito.mock(Processable.class); private SomeTask someTask = new SomeTask();public ServiceIntegrationTest() { initializeMocks(); }private void initializeMocks() { Mockito.doAnswer(new Answer<Object>() { public Object answer(InvocationOnMock invocation) throws Throwable { // Simulation of connection to external services LOGGER.debug("Sleeping"); Thread.sleep(5000); LOGGER.debug("Stopped Sleeping"); return null; } }).when(timeConsumingExternalServiceMock).process(); // Inject the mock to the Task - in any possible way someTask.setTimeConsumingExternalService(timeConsumingExternalServiceMock); }public void executeTest() { executorService.execute(someTask); }public static void main(String args[]) { ServiceIntegrationTest integrationTest = new ServiceIntegrationTest(); integrationTest.executeTest(); } } And the output to the console: 2012-10-07 22:42:37,378 DEBUG pl.grzejszczak.marcin.SomeTask:21 Before processing2012-10-07 22:42:37,389 DEBUG pl.grzejszczak.marcin.ServiceIntegrationTest:28 Sleeping2012-10-07 22:42:42,390 DEBUG pl.grzejszczak.marcin.ServiceIntegrationTest:30 Stopped Sleeping2012-10-07 22:42:42,392 DEBUG pl.grzejszczak.marcin.SomeTask:23 After processing Let’s take a closer look at the most important part in which an Answer for the execution of the service is being created Mockito.doAnswer(new Answer<Object>() { public Object answer(InvocationOnMock invocation) throws Throwable { // Simulation of connection to external services LOGGER.debug("Sleeping"); Thread.sleep(5000); LOGGER.debug("Stopped Sleeping"); return null; } }).when(timeConsumingExternalServiceMock).process(); This piece of code changes the default action that should be done by the given object on a given method execution. In this particular case we had to mock a method that returns void – that’s why we start with doAnswer(…) and finish with when(…).process(). That is how inside the integration test I managed to create a simulation of waiting for the service to finish. If you have any ideas or comments on how you would do it in another way please feel free to post a comment below   Reference: Simulation of time consuming actions in integration tests from our JCG partner Marcin Grzejszczak at the Blog for coding addicts blog. ...
software-development-2-logo

The Art of War: How it Applies to Software

Sun Tzu wrote: War is of vital importance to the state; hence it is a subject of inquiry which can on no account be neglected In our modern world, software is of vital importance to your organization. If you can build software consistently and reliably you will gain a tremendous advantage over your competition. If developing software is like waging war then who is the enemy? The enemy is complexity. Complexity comes from having to make dozens of     decisions correctly for your software project to even have a chance of succeeding. For every thing that you know at the outset of the project, there will be at least ten things that you don’t know. Complexity is compounded by having to make those decisions on a deadline, especially if that deadline is not well chosen. So when you consider launching a software project you must understand that you are looking at the tip of the iceberg. Your ability to handle uncertainty will dictate how successful you will be. Unlike a conventional war this enemy does not sleep, has no obvious weaknesses, and can not be deceived. Once you engage the enemy your success will depend on correctly estimating the magnitude of complexity and making excellent decisions. Sources of Uncertainty If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle. Attack by Stratagem, p. 18 Uncertainty comes from a few main sources:Team resources unfamiliar with the technologies to be used Requirements that are incomplete or inconsistent Technical requirements that turn out to be infeasible Inability to understand project dependencies Inability to formulate a correct and understandable planUnlike fighting a conventional enemy, complexity will find all your weaknesses. The only way to defeat complexity is through understanding the requirements, having well trained teams, building a solid project plan, and executing well. Unfortunately the way most organizations develop software resembles the Charge of the Light Brigade (poem). In that battle about 400 men on horses attacked 20 battalions of infantry supported by 50 artillery pieces. Needless to say it was a slaughter. The Art of War (online) Several principles outlined by Sun Tzu apply to software development as well:Deception Leading to advantage Energy Using Spies Strengths and weaknesses Winning wholeDeceptionSince the enemy is complexity we can not deceive it. Rather the problem is that we allow ourselves to be deceived by complexity. How often do you see developer’s say that they can code anything over a weekend over a case of Red Bull? People generally do not launch software projects because they want to fail;but when only 3 out of 10 projects succeed, it shows that people have been deceived about the complexity of building software. This statistic has haunted us for 50 years. Senior management should really pause and consider if all their ducks are lined up in a row before embarking on a software project. Unfortunately senior management is still underestimating complexity and sending teams to face virtually impossible projects. Leading to advantageThus it is that in war the victorious strategist seeks battle after the victory has been won, whereas he who is destined to defeat first fights and afterwards looks for victory in the midst of the fight. Tactical Dispositions, p. 15 Complexity emerges from the sources of uncertainty mentioned above. Successful organizations plan to remove known uncertainties and have a plan to handle the ones that will emerge. Getting into a project before understanding the magnitude of the uncertainty is a recipe for failure. EnergyThat the impact of your army may be like a grindstone dashed against an egg—this is effected by the science of weak points and strong Energy, p. 4 The team’s energy needs to be directed against uncertainty with appropriate force at the appropriate time. When this happens uncertainty will be minimized and the chances for success will increase. It is imperative to create a table of all risks that might affect your software project. If you aggressively minimize the probability of risks triggering you will reduce uncertainty and increase your chances of success. Software projects succeed when there is a rhythm to the attack on complexity.  High intensity problem solving needs to be followed by lower intensity stability building.  The team must move at a sustainable pace or risk burning out.  Software teams do not succeed when they are working 10+ hours a day; they become like dull swords — unable to do anything. Using Spies Foreknowledge cannot be elicited from ghosts and spirits; it cannot be inferred from comparison of previous events, or from the calculations of the heavens, but must be obtained from people who have knowledge of the enemy’s situation. Using Spies, 5 —6The enemy is complexity and is intangible, i.e. invisible, odorless, and untouchable. Your spies are your business analysts, architects, and project managers. Your business analysts will work with the business to define the scope of the complexity.  Ask for anything you want, but commit to build all you ask! Remember that all large complex systems that work are built from small simple systems that work, so aim to build the smallest usable product initially. Asking for too much and providing insufficient resources and/or time will lead to a failed project. The architects provide checks and balances on the business analysts to make sure that the project is feasible. The architects will provide key dependency information to the project managers who make sure that a proper execution plan is created and followed. Each of these spies sees a different aspect of complexity that is not visible to other people. Unless the reports of three types is combined effectively you risk not knowing the extent of the software that you are trying to build.  If you go into battle without proper intelligence you are back to the scenario of the Charge of the Light Brigade. Strengths and weaknessesMilitary tactics are like unto water; for water in its natural course runs away from high places and hastens downwards.  So in war, the way is to avoid what is strong and to strike at what is weak. Weak Points and Strong, p. 29—30 Water exhibits ordered flexibility. It is ordered because it seeks to flow downhill; however, it is flexible and will go around rocks and other obstacles. A software project needs to make continual progress without getting sandbagged with obstacles. Methodologies like RUP or Agile software development can make sure that you exhibit ordered flexibility. Winning whole Winning whole in a software project means delivering the software on time and on budget without destroying the health and reputations of your team (including management). Failed projects extend their effects to every member of the team and everyone’s resume. When you engage in actual fighting, if victory is long in coming, then men’s weapons will grow dull and their ardor will be damped. Waging War, p. 3When organizations bite off more than they can chew they exert tremendous pressures on the team resources to work extended hours to make deadlines that are often unrealistic. In the pressure cooker you can expect key personnel to defect and put you into a worse position. How many times have you found yourself on a Death March? Conclusion Both waging war and software development are serious topics that involve important struggles. If software development is a war against ignorance, uncertainty, and complexity then many of the strategies and tactics outlined in The Art of War give us direction on how to execute a successful project.   Reference: The Art of War: How it Applies to Software from our JCG partner Dalip Mahal at the Accelerated Development blog. ...
software-development-2-logo

A Theorical Look into Object/Resource Pool Pattern

Definition: Sometimes our software projects need costly object creations (database connections, socket connections, large graphical objects etc.). This cost may be about time and/or memory. Those objects may also be needed to be created frequently. At that moment, Object/Resource Pool pattern comes to help.         Object Pool pattern:Reuses unused costly objects without re-creating those objects (e.g. books in a library). May create objects eagerly before they are needed, to manage application performance. May limit created number of objects to manage application performance and user access.Working principle:Client requests an object (Reusable) from object pool. Object pool has a list of predefined reusables and gives an available one from pool. For the client’s point of view, given reusable is a new object but it is probably (object creation strategies will be told later) a pre-created object with a new object’s field values. For the optimum performance, client should notify object pool when reusable is no longer needed. Just in case client doesn’t notify, object pool may define a timeout for each created reusable. Object pool must also have a synchronization mechanism for reusable serving, especially on multithreaded or multiuser applications. When to use/Trade offs: If objects are not costly, object pool pattern should not be used because this pattern needs clients to inform object pool (brings extra code complexity) and object creation management code (synchronization of pool, object creation limitation strategy etc.). This extra management code also brings a little performance loss. If performance gain of reusable object usage is bigger than extra management code performance loss, and application must frequently create those reusable objects, object pool pattern is recommended to be used. Implementation Strategies: Object pool limit:If memory is limited and/or maximum number of clients are needed to be restricted, an object number limit value may be defined for object pool (limited pool). If there is no restriction, limit value is not needed (unlimited pool). If all object pool list is in use, new objects may be created (expanding pool) or client may be enforced to waiting (fixed pool).Eager/lazy creation:If application startup is performed infrequently and startup time is not very important, object pool list objects may be created at the startup (eager ceration). If no restrictions exist, no objects are needed to be created eagerly (lazy creation). Some number of objects may be created eagerly, others may be created lazily according to the application parameters (mixed creation).Empty object pool strategy:If object pool is expanding, create a new object and return it. If object pool is not expanding, return null or enforce client to wait until an object is ready to be given.Synchronization strategy:Object pool should have a synchronization mechanism for object serving. Otherwise, especially on multiuser systems, pooling system may fail.Unused object strategy:Usage of the pool objects may be lower than expected in some cases. Object pool may have a pool object removing strategy (i.e. descreasing object limit) for increasing performance.Returning object strategy:Clients should return pool objects back to the pool after their job is completed. But pool code can’t control this and poor client code may return pool objects. For this situation, object pool may have a timer mechanism for given but unused pool objects.According to the project requirements, some of those strategies or mixed versions should be selected and implemented for object pool. But “reusing available objects” principle is the key and can not be changed for any implementation strategy.   Reference: A Theorical Look into Object/Resource Pool Pattern from our JCG partner Cagdas Basaraner at the CodeBuild blog. ...
software-development-2-logo

How many classes and packages should you have?

Real world experience. Once upon a time there lived a family of programming bears. Each morning mammy bear, daddy bear and baby bear would rise and eat breakfast together in their secluded cottage deep in the forest before whipping out their three laptops to spend hours cheerily writing Java code. One day, while the bears were out shopping, professor of automation theory and software engineering, Maria G. Locks, stumbled upon the cottage while lost in the woods. Cupping her hands against the glass as she peered through a window she spied the glowing screens of three laptops and immediately smashed her way inside.   She sat at the first laptop, cracked its password and pored over the Java source code. It was a frightful mess. With just a few, monstrous classes containing thousand-line-long functions, the structure gashed her soul. Distraught, she moved to the second laptop and cracked its password. A rancid splurge of code vomited itself up onto the screen, this time with hundreds of fiddly packages hosting hundreds of tiny classes, each in turn baring masses of disjointed functions. Cracking a final password she sank weeping before the third laptop. This code was beautiful. The packages were neither too big nor too small, neither too many nor too few. The well-proportioned classes housed coherent implementation detail. The functions, appropriately information-hidden, performed single tasks without fuss. Then the three bears burst in through the front door and daddy bear roared with murderous rage at the sight of the intruder. Professor Locks, screaming, leaped to her feet. Mammy bear shrieked something about porridge and a bed but no one could really remember how the tale ended so the whole thing was just abandoned. The point being? When is our software neither over-encapsulated, with too many packages and classes, nor under-encapsulated, with too few? When is it just right? Let us try to answer these questions by modeling a Java program. Novices often ask, “How many classes should you have?” Though unaware of their omission, they are actually asking, “How many classes should I have to satisfy a particular constraint?” We, too, must choose our constraint. Real-world constraints usually relate to the problem-domain but, wishing our model to be as widely-applicable as it is simple, we shall choose as our constraint that we want to minimize the number of potential dependencies that our system can support. That is, we wish to minimize the system’s potential coupling. Though simple, our choice is not without merit, for we wish (deep breath) to minimize the cost of developing our software and that means minimizing ripple-effect updates by minimizing the number of dependencies involved and that means minimizing the conditions that encourage dependency-formation and that means minimizing the number of potential dependencies. Eyebrows no doubt rise at such rattling chains of logic but the prize – that of an objective measure of how many functions, classes and packages we should have – might be worth some leeway. Model assumptions. Our model is based on two assumptions: one sensible, one completely bonkers. The first assumption is that our system consists of n functions. Our task is find the number classes and packages over which we should distribute these n functions so that we minimize the potential coupling. The second assumption is that all functions will be evenly distributed over all classes and packages, that each class will have d functions visible to other classes in the package and that each package will have the same number, d, functions visible to other classes in other packages. See? Bonkers. But not without a smidgen of rationale. We know that evenly-distributing functions minimizes potential coupling in almost all cases. Because potential coupling rises with the square of number of elements that can see one another, if a class has twice as many functions then it will have four times as much potential coupling. As for having a strict number of default-accessor functions per class and public functions per package, well, precisely no one designs a system that way. Programmers try, though, to minimize the number of maximally-visible elements so we can look at d = 1 as an ideal if asymptotic goal. Certainly once a system is up and running it must have an average number of default-accessor functions per class and public functions per package. Given that we want both numbers to be small it is perhaps not so outlandish to find that they might be similar. Our assumptions, however, these are. Judge their plausibility for yourself. Finally, let us label our quarry: we wish to know how many packages, pa, and classes, r, our system should have. So, how do we find pa and r? What a function sees. Let us take any function – we shall call it test1() – in our hypothetical, evenly-distributed system and ask: how many other functions can test1() see? Everything here follows from this question. There are three groups of functions that test1() can see. It can see functions within its own class. It can see functions in other classes within its own package. And it can see functions in other classes in other packages. These three groups, taken together, comprise all the functions visible to test1(), that is, they give the potential coupling of test1(). For reasons that will become clear, we wish to find an equation for this potential coupling. To do this we must find three terms, one corresponding to each of these three groups. Functions in own class.Figure 1 shows three packages, each with two classes, each class housing three functions. The red functions are public (their classes are also public), the green private and default-accessor functions are blue. Our test1() function is selected at random to be the one in the bottom-left, as indicated throughout. How many functions can test1() see in its own class? Well, it can see all of them; in the example above it can see both other functions in its class. We know that the number of functions in any class is just the total number of functions, n, divided by the total number of classes, r, so we might think that the number is n / r. But this would also include a dependency from test1() on itself, which we would like to ignore, thus the number of dependencies that can form is one less than the number of functions in the class, that is:Functions in own package.How many functions can test1() see in other classes within its own package? It can see those functions that are not private in all other classes in the package; in the example above it can see just one other public function. We know that the number of functions that are not private in each class is d by definition. And we know that the number of classes per package is just the total number of classes, r, divided by the number of packages, pa, though, again, we wish to exclude to possibility of test1()‘s forming dependencies on its own class. So the number of functions it can see in its own package is:Functions in other packages.How many functions can test1() see in other packages? It can see all public functions in all public classes in all other packages; the example above shows it forming dependencies on two other public functions. We know that the number of packages is pa and the number of public functions per package is d by definition. Again, we shall discount forming dependencies within its own package, so the number of functions it can see in other packages is:These are the three terms that make up the potential coupling of our one function. To find the potential coupling of the entire system we just multiply them by the total number of functions, n, to give our final equation for the potential coupling, s:Now, it is well-known that the greatest complaint raised by students learning Java is that there are too few partial differential equations. It so happens that our quest to find the number of classes and packages to minimize potential coupling involves just such beasts: for we face an optimization problem. Remember those from high-school? Set the derivative to zero and solve? That’s all we need to do now, thank you, Mister Newton … erh, Herr Leibniz … whatever. To find the number of classes, r, that minimizes the potential coupling, s, we calculate the derivative of s with respect to r:To find the number of packages, pa, that minimizes the potential coupling, s, we calculate the derivative of s with respect to pa:Solving both simultaneous equations gives solutions:Thus if you have a system of 400 functions, that is n = 400, and aim to have 5 functions per class visible within a package, that is d = 5, then the number of packages you should have is 4, each with 18 classes. Figure 4 plots our model’s potential coupling of 400 functions over all possible of numbers of packages, on the x-axis, and numbers of classes, on the y-axis, with each coordinate coloured red in proportion to the amount of potential coupling generated by that combination of package and class number; thus light pink implies wonderfully-low potential coupling. (The graph is triangular as we cannot have fewer classes than non-empty packages.) The black square indicates the potential coupling minimum.Of course, you would never create such a system. Having only 4 packages for so many functions strikes as far too few; brutal reality would surely barge in and dictate 8 packages, or 10. This is where the efficiency* of your system comes into play. 100%-efficient systems are as rare as 100%-efficient steam engines. You should gladly descend to 80% or 70% efficiency to win some elbow-room for your design, to enjoy more semantically-driven packages and less-regular distributions than the restrictive ideal case demands. But you may hesitate to fall to the crass encapsulation of only 40% or 30% efficiency, opening your system to the possibility of enormous numbers of dependencies and their subsequent ripple-effect costs. Infra-red-goggles time. Figure 5 shows an efficiency heat-map of our 400 functions generated by the same potential coupling equation derived above. The dark-blue area shows efficiency of above 90%, a tropical island untrodden by human feet. The cyan, green and yellow bands offer configurations of a more reasonable trade-off between practicality and potential expense. The red area, well …Summary How many classes and packages should you have? Altogether now, “It depends.” It depends in particular on the constraint you choose to satisfy. The constraint chosen here was that which tries to minimize development cost by minimizing potential ripple-effect updates. A simple model was built on some dubious assumptions which lead to some perhaps interesting conclusions.The model can easily be extended, at the cost of complexity, to more-accurately capture reality though, in the end, the model itself has perhaps less to offer than the method by which it was conceived. However many classes and packages you decide to have, at least be aware of the principles by which you make your decisions. The only bad design is no design. Notes: * For the absolute ideal efficiency, and as d=1   Reference: How many classes and packages should you have? from our JCG partner Edmund Kirwan at the A blog about software. blog. ...
spring-interview-questions-answers

Spring AOP in security – controlling creation of UI components via aspects

The following post will show how in one of the projects that I took part in we used Spring’s AOP to introduce some security related functionalities. The concept was such that in order for the user to see some UI components he needed to have a certain level of security privillages. If that requirement was not met then the UIComponent was not presented. Let’s take a look at the project structure:  Then there were also the aopApplicationContext.xml : <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:context="http://www.springframework.org/schema/context" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd http://www.springframework.org/schema/tx http://www.springframework.org/schema/tx/spring-tx-3.0.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.1.xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-3.0.xsd"><aop:aspectj-autoproxy /> <context:annotation-config /> <context:component-scan base-package="pl.grzejszczak.marcin.aop"> <context:exclude-filter type="annotation" expression="org.aspectj.lang.annotation.Aspect"/> </context:component-scan> <bean class="pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor" factory-method="aspectOf"/></beans> Now let’s take a look at the most interesting lines of the Spring’s application context. First we have all the required schemas – I don’t think that this needs to be explained in more depth. Then we have: <aop:aspectj-autoproxy/> which enables the @AspectJ support. Next there is the <context:annotation-config /> <context:component-scan base-package="pl.grzejszczak.marcin.aop"> <context:exclude-filter type="annotation" expression="org.aspectj.lang.annotation.Aspect"/> </context:component-scan> first we are turning on Spring configuration via annotations. Then deliberatly we exclude aspects from being initialized as beans by Spring itself. Why? Because… <bean class="pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor" factory-method="aspectOf"/> we want to create the aspect by ourselves and provide the factory-method=”aspectOf” . By doing so our aspect will be included in the autowiring process of our beans – thus all the fields annotated with the @Autowired annotation will get the beans injected. Now let’s move on to the code: UserServiceImpl.java package pl.grzejszczak.marcin.aop.service;import org.springframework.stereotype.Service;import pl.grzejszczak.marcin.aop.type.Role; import pl.grzejszczak.marcin.aop.user.UserHolder;@Service public class UserServiceImpl implements UserService { private UserHolder userHolder;@Override public UserHolder getCurrentUser() { return userHolder; }@Override public void setCurrentUser(UserHolder userHolder) { this.userHolder = userHolder; }@Override public Role getUserRole() { if (userHolder == null) { return null; } return userHolder.getUserRole(); } } The class UserServiceImpl is immitating a service that would get the current user information from the db or from the current application context. UserHolder.java package pl.grzejszczak.marcin.aop.user;import pl.grzejszczak.marcin.aop.type.Role;public class UserHolder { private Role userRole;public UserHolder(Role userRole) { this.userRole = userRole; }public Role getUserRole() { return userRole; }public void setUserRole(Role userRole) { this.userRole = userRole; } } This is a simple holder class that holds information about current user Role. Role.java package pl.grzejszczak.marcin.aop.type;public enum Role { ADMIN("ADM"), WRITER("WRT"), GUEST("GST");private String name;private Role(String name) { this.name = name; }public static Role getRoleByName(String name) {for (Role role : Role.values()) {if (role.name.equals(name)) { return role; } }throw new IllegalArgumentException("No such role exists [" + name + "]"); }public String getName() { return this.name; }@Override public String toString() { return name; } } Role is an enum that defines a role for a person being an Admin, Writer or a Guest. UIComponent.java package pl.grzejszczak.marcin.aop.ui;public abstract class UIComponent { protected String componentName;protected String getComponentName() { return componentName; }} An abstraction over concrete implementations of some UI components. SomeComponentForAdminAndGuest.java package pl.grzejszczak.marcin.aop.ui;import pl.grzejszczak.marcin.aop.annotation.SecurityAnnotation; import pl.grzejszczak.marcin.aop.type.Role;@SecurityAnnotation(allowedRole = { Role.ADMIN, Role.GUEST }) public class SomeComponentForAdminAndGuest extends UIComponent {public SomeComponentForAdminAndGuest() { this.componentName = "SomeComponentForAdmin"; }public static UIComponent getComponent() { return new SomeComponentForAdminAndGuest(); } } This component is an example of a UI component extention that can be seen only by users who have roles of Admin or Guest. SecurityAnnotation.java package pl.grzejszczak.marcin.aop.annotation;import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy;import pl.grzejszczak.marcin.aop.type.Role;@Retention(RetentionPolicy.RUNTIME) public @interface SecurityAnnotation { Role[] allowedRole(); } Annotation that defines a roles that can have this component created. UIFactoryImpl.java package pl.grzejszczak.marcin.aop.ui;import org.apache.commons.lang.NullArgumentException; import org.springframework.stereotype.Component;@Component public class UIFactoryImpl implements UIFactory {@Override public UIComponent createComponent(Class<? extends UIComponent> componentClass) throws Exception { if (componentClass == null) { throw new NullArgumentException("Provide class for the component"); } return (UIComponent) Class.forName(componentClass.getName()).newInstance(); } } A factory class that given the class of an object that extends UIComponent returns a new instance of the given UIComponent. SecurityInterceptor.java package pl.grzejszczak.marcin.aop.interceptor;import java.lang.annotation.Annotation; import java.lang.reflect.AnnotatedElement; import java.util.Arrays; import java.util.List;import org.aspectj.lang.ProceedingJoinPoint; import org.aspectj.lang.annotation.Around; import org.aspectj.lang.annotation.Aspect; import org.aspectj.lang.annotation.Pointcut; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Autowired;import pl.grzejszczak.marcin.aop.annotation.SecurityAnnotation; import pl.grzejszczak.marcin.aop.service.UserService; import pl.grzejszczak.marcin.aop.type.Role; import pl.grzejszczak.marcin.aop.ui.UIComponent;@Aspect public class SecurityInterceptor { private static final Logger LOGGER = LoggerFactory.getLogger(SecurityInterceptor.class);public SecurityInterceptor() { LOGGER.debug("Security Interceptor created"); }@Autowired private UserService userService;@Pointcut("execution(pl.grzejszczak.marcin.aop.ui.UIComponent pl.grzejszczak.marcin.aop.ui.UIFactory.createComponent(..))") private void getComponent(ProceedingJoinPoint thisJoinPoint) { }@Around("getComponent(thisJoinPoint)") public UIComponent checkSecurity(ProceedingJoinPoint thisJoinPoint) throws Throwable { LOGGER.info("Intercepting creation of a component");Object[] arguments = thisJoinPoint.getArgs(); if (arguments.length == 0) { return null; }Annotation annotation = checkTheAnnotation(arguments); boolean securityAnnotationPresent = (annotation != null);if (securityAnnotationPresent) { boolean userHasRole = verifyRole(annotation); if (!userHasRole) { LOGGER.info("Current user doesn't have permission to have this component created"); return null; } } LOGGER.info("Current user has required permissions for creating a component"); return (UIComponent) thisJoinPoint.proceed(); }/** * Basing on the method's argument check if the class is annotataed with * {@link SecurityAnnotation} * * @param arguments * @return */ private Annotation checkTheAnnotation(Object[] arguments) { Object concreteClass = arguments[0]; LOGGER.info("Argument's class - [{}]", new Object[] { arguments }); AnnotatedElement annotatedElement = (AnnotatedElement) concreteClass; Annotation annotation = annotatedElement.getAnnotation(SecurityAnnotation.class); LOGGER.info("Annotation present - [{}]", new Object[] { annotation }); return annotation; }/** * The function verifies if the current user has sufficient privilages to * have the component built * * @param annotation * @return */ private boolean verifyRole(Annotation annotation) { LOGGER.info("Security annotation is present so checking if the user can use it"); SecurityAnnotation annotationRule = (SecurityAnnotation) annotation; List<Role> requiredRolesList = Arrays.asList(annotationRule.allowedRole()); Role userRole = userService.getUserRole(); return requiredRolesList.contains(userRole); } } This is the aspect defined at the pointcut of executing a function createComponent of the UIFactory interface. Inside the Around advice there is the logic that first checks what kind of an argument has been passed to the method createComponent (for example SomeComponentForAdminAndGuest.class). Next it is checking if this class is annotated with SecurityAnnotation and if that is the case it checks what kind of Roles are required to have the component created. Afterwards it checks if the current user (from UserService to UserHolder’s Roles) has the required role to present the component. If that is the case thisJoinPoint.proceed() is called which in effect returns the object of the class that extends UIComponent. Now let’s test it – here comes the SpringJUnit4ClassRunner AopTest.java package pl.grzejszczak.marcin.aop;import org.junit.Assert; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;import pl.grzejszczak.marcin.aop.service.UserService; import pl.grzejszczak.marcin.aop.type.Role; import pl.grzejszczak.marcin.aop.ui.SomeComponentForAdmin; import pl.grzejszczak.marcin.aop.ui.SomeComponentForAdminAndGuest; import pl.grzejszczak.marcin.aop.ui.SomeComponentForGuest; import pl.grzejszczak.marcin.aop.ui.SomeComponentForWriter; import pl.grzejszczak.marcin.aop.ui.UIFactory; import pl.grzejszczak.marcin.aop.user.UserHolder;@RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(locations = { "classpath:aopApplicationContext.xml" }) public class AopTest {@Autowired private UIFactory uiFactory;@Autowired private UserService userService;@Test public void adminTest() throws Exception { userService.setCurrentUser(new UserHolder(Role.ADMIN)); Assert.assertNotNull(uiFactory.createComponent(SomeComponentForAdmin.class)); Assert.assertNotNull(uiFactory.createComponent(SomeComponentForAdminAndGuest.class)); Assert.assertNull(uiFactory.createComponent(SomeComponentForGuest.class)); Assert.assertNull(uiFactory.createComponent(SomeComponentForWriter.class)); } } And the logs: pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:26 Security Interceptor created pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:38 Intercepting creation of a component pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:48 Argument's class - [[class pl.grzejszczak.marcin.aop.ui.SomeComponentForAdmin]] pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:54 Annotation present - [@pl.grzejszczak.marcin.aop.annotation.SecurityAnnotation(allowedRole=[ADM])] pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:57 Security annotation is present so checking if the user can use it pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:70 Current user has required permissions for creating a component pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:38 Intercepting creation of a component pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:48 Argument's class - [[class pl.grzejszczak.marcin.aop.ui.SomeComponentForAdminAndGuest]] pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:54 Annotation present - [@pl.grzejszczak.marcin.aop.annotation.SecurityAnnotation(allowedRole=[ADM, GST])] pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:57 Security annotation is present so checking if the user can use it pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:70 Current user has required permissions for creating a component pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:38 Intercepting creation of a component pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:48 Argument's class - [[class pl.grzejszczak.marcin.aop.ui.SomeComponentForGuest]] pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:54 Annotation present - [@pl.grzejszczak.marcin.aop.annotation.SecurityAnnotation(allowedRole=[GST])] pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:57 Security annotation is present so checking if the user can use it pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:66 Current user doesn't have permission to have this component created pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:38 Intercepting creation of a component pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:48 Argument's class - [[class pl.grzejszczak.marcin.aop.ui.SomeComponentForWriter]] pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:54 Annotation present - [@pl.grzejszczak.marcin.aop.annotation.SecurityAnnotation(allowedRole=[WRT])] pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:57 Security annotation is present so checking if the user can use it pl.grzejszczak.marcin.aop.interceptor.SecurityInterceptor:66 Current user doesn't have permission to have this component created The unit test shows that for given Admin role only first two components get created whereas for the two others nulls are returned (due to the fact that user doesn’t have proper rights). That is how in our project we used Spring’s AOP to create a simple framework that would check if the user can have the given component created or not. Thanks to this after having programmed the aspects one doesn’t have to remember about writing any security related code since it will be done for him.   Reference: Spring AOP in security – controlling creation of UI components via aspects from our JCG partner Marcin Grzejszczak at the Blog for coding addicts blog. ...
jcg-logo

What is Big Data – Theory to Implementation

What is Big Data? You may ask; and more importantly why it is the latest trend in nearly every business domain? Is it just a hype or its here to stay? As a matter of fact “Big Data” is a pretty straightforward term – its just what its says – a very large data-set. How large? The exact answer is “as large as you can imagine”! How can this data-set be so massively big? Because the data may come from everywhere and in enormous rates: RFID sensors that gather traffic data, sensors used to gather weather information, GPRS packets from cell phones, posts to social media sites, digital pictures and videos, online purchase transaction records, you name it! Big Data is an enormous data-set that may contain information from every possible source that produces data that we are interested in. Nevertheless Big Data is more than simply a matter of size; it is an opportunity to find insights in new and emerging types of data and content, to make businesses more agile, and to answer questions that were previously considered beyond our reach. That is why Big Data is characterized by four main aspects: Volume, Variety, Velocity, and Veracity(Value) known as “the four Vs of Big Data”. Let’s briefly examine what each one of them stands for and what challenges it presents: Volume Volume references the amount of content a business must be able to capture, store and access. 90% of the world’s data has been generated in the past two years alone. Organizations today are overwhelmed with volumes of data, easily amassing terabytes—even petabytes—of information of all types, some of which needs to be organized, secured and analyzed. Variety 80% of the world’s data is semi – structured. Sensors, smart devices and social media are generating this data through Web pages, weblog files, social-media forums, audio, video, click streams, e-mails, documents, sensor systems and so on. Traditional analytics solutions work very well with structured information, for example data in a relational database with a well formed schema. Variety in data types represents a fundamental shift in the way data is stored and analysis needs to be done to support today’s decision-making and insight process. Thus Variety represents the various types of data that can’t easily be captured and managed in a traditional relational database but can be easily stored and analyzed with Big Data technologies. Velocity Velocity requires analyzing data in near real time, aka “sometimes 2 minutes is too late!”. Gaining a competitive edge means identifying a trend or opportunity in minutes or even seconds before your competitor does. Another example is time-sensitive processes such as catching fraud where information must be analyzed as it streams into your enterprise in order to maximize its value. Time-sensitive data has a very short shelf-life; compelling organizations to analyze them in near real-time. Veracity (Value) Acting on data is how we create opportunities and derive value. Data is all about supporting decisions, so when you are looking at decisions that can have a major impact on your business, you are going to want as much information as possible to support your case. Nevertheless the volume of data alone does not provide enough trust for decision makers to act upon information. The truthfulness and quality of data is the most important frontier to fuel new insights and ideas. Thus establishing trust in Big Data solutions probably presents the biggest challenge one should overcome to introduce a solid foundation for successful decision making. While the existing installed base of business intelligence and data warehouse solutions weren’t engineered to support the four V’s, big data solutions are being developed to address these challenges. What follows is a brief presentation of the major open-source Java based tools that are available today and support Big Data :HDFS is the primary distributed storage used by Hadoop applications. A HDFS cluster primarily consists of a NameNode that manages the file system metadata and DataNodes that store the actual data. HDFS is specifically designed for storing vast amount of data, so it is optimized for storing/accessing a relatively small number of very large files compared to traditional file systems where are optimized to handle large numbers of relatively small files.Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.Apache HBase is the Hadoop database, a distributed, scalable, big data store. It provides random, realtime read/write access to Big Data and is optimized for hosting very large tables — billions of rows X millions of columns — atop clusters of commodity hardware. In its core Apache HBase is a distributed, versioned, column-oriented store modeled after Google’s Bigtable: A Distributed Storage System for Structured Data by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, Apache HBase provides Bigtable-like capabilities on top of Hadoop and HDFS.The Apache Cassandra is a performant, linear scalable and high available database that can run on commodity hardware or cloud infrastructure making it the perfect platform for mission-critical data. Cassandra’s support for replicating across multiple datacenters is best-in-class, providing lower latency for users and the peace of mind of knowing that you can survive regional outages. Cassandra’s data model offers the convenience of column indexes with the performance of log-structured updates, strong support for denormalization and materialized views, and powerful built-in caching.Apache Hive is a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. At the same time this language also allows traditional map/reduce programmers to plug in their custom mappers and reducers when it is inconvenient or inefficient to express this logic in HiveQL.Apache Pig is a platform for analyzing large data sets. It consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs. The salient property of Pig programs is that their structure is amenable to substantial parallelization, which in turns enables them to handle very large data sets. Pig’s infrastructure layer consists of a compiler that produces sequences of Map-Reduce programs. Pig’s language layer currently consists of a textual language called Pig Latin, which is developed with ease of programming, optimization opportunities and extensibility in mind.Apache Chukwa is an open source data collection system for monitoring large distributed systems. It is built on top of the Hadoop Distributed File System (HDFS) and Map/Reduce framework and inherits Hadoop’s scalability and robustness. Chukwa also includes a flexible and powerful toolkit for displaying, monitoring and analyzing results to make the best use of the collected data.Apache Ambari is a web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters which includes support for Hadoop HDFS, Hadoop MapReduce, Hive, HCatalog, HBase, ZooKeeper, Oozie, Pig and Sqoop. Ambari also provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually alongwith features to diagnose their performance characteristics in a user-friendly manner.Apache ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. In short Apache ZooKeeper is a high-performance coordination service for distributed applications like those run on a hadoop cluster.Apache Sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases.Apache Oozie is a scalable, reliable and extensible workflow scheduler system to manage Apache Hadoop jobs. Oozie Workflow jobs are Directed Acyclical Graphs (DAGs) of actions. Oozie Coordinator jobs are recurrent Oozie Workflow jobs triggered by time (frequency) and data availabilty. Oozie is integrated with the rest of the Hadoop stack supporting several types of Hadoop jobs out of the box (such as Java map-reduce, Streaming map-reduce, Pig, Hive, Sqoop and Distcp) as well as system specific jobs (such as Java programs and shell scripts).Apache Mahout is a scalable machine learning and data mining library. Currently Mahout supports mainly four use cases:Recommendation mining : Takes users’ behavior and from that tries to find items users might like. Clustering : Takes e.g. text documents and groups them into groups of topically related documents. Classification : Learns from existing categorized documents what documents of a specific category look like and is able to assign unlabeled documents to the (hopefully) correct category. Frequent itemset mining : Takes a set of item groups (terms in a query session, shopping cart content) and identifies, which individual items usually appear together.Apache HCatalog is a table and storage management service for data created using Apache Hadoop. This includes:Providing a shared schema and data type mechanism. Providing a table abstraction so that users need not be concerned with where or how their data is stored. Providing interoperability across data processing tools such as Pig, Map Reduce, and Hive.  That’s it; Big Data, a short theoretical introduction and a compact matrix of implementation approaches focused on overcoming the problems of a new era – the era that forces us to ask bigger questions!   Happy Coding Byron ...
xstream-logo

XStream – XStreamely easy way to work with XML data in Java

From time to time there is a moment when we have to deal with XML data. And most of the time it is not the happiest day in our life. There is even a term “XML hell” describing situation when programmer has to deal with many XML configuration files that are hard to comprehend. But, like it or not, sometimes we have no choice, mostly because specification from client says something like “use configuration written in XML file” or something similar. And in such cases, XStream comes with its very cool features that make dealing with XML really less painful.Overview XStream is a small library to serialize data between Java objects and XML. It’s lightweight, small, has nice API and what is most important, it works with and without custom annotations that we might be not allowed to add when we are not the owner of Java classes. First example Suppose we have a requirement to load configuration from xml file: <config> <inputFile>/Users/tomek/work/mystuff/input.csv</inputFile> <truststoreFile>/Users/tomek/work/mystuff/truststore.ts</truststoreFile> <keystoreFile>/Users/tomek/work/mystuff/CN-user.jks</keystoreFile><!-- ssl stores passwords--> <truststorePassword>password</truststorePassword> <keystorePassword>password</keystorePassword><!-- user credentials --> <user>user</user> <password>secret</password> </config> And we want to load it into Configuration object: public class Configuration {private String inputFile; private String user; private String password;private String truststoreFile; private String keystoreFile; private String keystorePassword; private String truststorePassword;// getters, setters, etc. } So basically what we have to do is: FileReader fileReader = new FileReader("config.xml"); // load our xml file XStream xstream = new XStream(); // init XStream // define root alias so XStream knows which element and which class are equivalent xstream.alias("config", Configuration.class); Configuration loadedConfig = (Configuration) xstream.fromXML(fileReader); And that’s all, easy peasy Something more serious Ok, but previous example is very basic so now let’s do something more complicated: real XML returned by real WebService. <DATA xmlns=""> <BAN> <UPDATED_AT>2013-03-09</UPDATED_AT> <TROUBLEMAKER> <NAME1>JOHN</NAME1> <NAME2>EXAMPLE</NAME2> <AGE>24</AGE> <NUMBER>ASD123123</NUMBER> </TROUBLEMAKER> </BAN> <BAN> <UPDATED_AT>2012-03-10</UPDATED_AT> <TROUBLEMAKER> <NAME1>ANNA</NAME1> <NAME2>BAKER</NAME2> <AGE>26</AGE> <NUMBER>AXN567890</NUMBER> </TROUBLEMAKER> </BAN> <BAN> <UPDATED_AT>2010-12-05</UPDATED_AT> <TROUBLEMAKER> <NAME1>TOM</NAME1> <NAME2>MEADOW</NAME2> <NUMBER>SGH08945</NUMBER> <AGE>48</AGE> </TROUBLEMAKER> </BAN> </DATA> What we have here is simple list of bans written in XML. We want to load it into collection of Ban objects. So let’s prepare some classes (getters/setters/toString omitted): public class Data { private List bans = new ArrayList(); }public class Ban { private String dateOfUpdate; private Person person; }public class Person { private String firstName; private String lastName; private int age; private String documentNumber; } As you can see there is some naming and type mismatch between XML and Java classes (e.g. field name1->firstName, dateOfUpdate is String not a Date), but it’s here for some example purposes. So the goal here is to parse XML and get Data object with populated collection of Ban instances containing correct data. Let’s see how it can be achieved. Parse with annotations First, easier way is to use annotations. And that’s the suggested approach in situation when we can modify Java classes to which XML will be mapped. So we have: @XStreamAlias("DATA") // maps DATA element in XML to this class public class Data {// Here is something more complicated. If we have list of elements that are // not wrapped in a element representing a list (like we have in our XML: // multiple <ban> elements not wrapped inside <bans> collection, // we have to declare that we want to treat these elements as an implicit list // so they can be converted to List of objects. @XStreamImplicit(itemFieldName = "ban") private List bans = new ArrayList(); }@XStreamAlias("BAN") // another mapping public class Ban {/* We want to have different field names in Java classes so we define what element should be mapped to each field */ @XStreamAlias("UPDATED_AT") // private String dateOfUpdate;@XStreamAlias("TROUBLEMAKER") private Person person; }@XStreamAlias("TROUBLEMAKER") public class Person {@XStreamAlias("NAME1") private String firstName;@XStreamAlias("NAME2") private String lastName;@XStreamAlias("AGE") // String will be auto converted to int value private int age;@XStreamAlias("NUMBER") private String documentNumber; And actual parsing logic is very short: FileReader reader = new FileReader("file.xml"); // load fileXStream xstream = new XStream(); xstream.processAnnotations(Data.class); // inform XStream to parse annotations in Data class xstream.processAnnotations(Ban.class); // and in two other classes... xstream.processAnnotations(Person.class); // we use for mappings Data data = (Data) xstream.fromXML(reader); // parse// Print some data to console to see if results are correct System.out.println("Number of bans = " + data.getBans().size()); Ban firstBan = data.getBans().get(0); System.out.println("First ban = " + firstBan.toString()); As you can see annotations are very easy to use and as a result final code is very concise. But what to do in situation when we can’t modify mapping classes? We can use different approach that doesn’t require any modifications in Java classes representing XML data. Parse without annotations When we can’t enrich our model classes with annotations, there is another solution. We can define all mapping details using methods from XStream object: FileReader reader = new FileReader("file.xml"); // three first lines are easy, XStream xstream = new XStream(); // same initialisation as in the xstream.alias("DATA", Data.class); // basic example abovexstream.alias("BAN", Ban.class); // two more aliases to map... xstream.alias("TROUBLEMAKER", Person.class); // between node names and classes// We want to have different field names in Java classes so // we have to use aliasField(<fieldInXml>, <mappedJavaClass>, <mappedFieldInJavaClass>) xstream.aliasField("UPDATED_AT", Ban.class, "dateOfUpdate"); xstream.aliasField("TROUBLEMAKER", Ban.class, "person");xstream.aliasField("NAME1", Person.class, "firstName"); xstream.aliasField("NAME2", Person.class, "lastName"); xstream.aliasField("AGE", Person.class, "age"); // notice here that XML will be auto-converted to int "age" xstream.aliasField("NUMBER", Person.class, "documentNumber");/* Another way to define implicit collection */ xstream.addImplicitCollection(Bans.class, "bans");Data data = (Data) xstream.fromXML(reader); // do the actual parsing// let's print results to check if data was parsed System.out.println("Number of bans = " + data.getBans().size()); Ban firstBan = data.getBans().get(0); System.out.println("First ban = " + firstBan.toString()); As you can see XStream allows to easily convert more complicated XML structures into Java objects, it also gives a possibility to tune results by using different names if this from XML doesn’t suit our needs. But there is one thing should catch your attention: we are converting XML representing a Date into raw String which isn’t quite what we would like to get as a result. That’s why we will add converter to do some job for us. Using existing custom type converter XStream library comes with set of built converters for most common use cases. We will use DateConverter. So now our class for Ban looks like that: public class Ban {private Date dateOfUpdate; private Person person; } And to use DateConverter we simply have to register it with date format that we expect to appear in XML data: xstream.registerConverter(new DateConverter("yyyy-MM-dd", new String[] {})); and that’s it. Now instead of String our object is populated with Date instance. Cool and easy! But what about classes and situations that aren’t covered by existing converters? We could write our own. Writing custom converter from scratch Assume that instead of dateOfUpdate we want to know how many days ago update was done: public class Ban { private int daysAgo; private Person person; } Of course we could calculate it manually for each Ban object but using converter that will do this job for us looks more interesting. Our DaysAgoConverter must implement Converter interface so we have to implement three methods with signatures looking a little bit scary: public class DaysAgoConverter implements Converter { @Override public void marshal(Object source, HierarchicalStreamWriter writer, MarshallingContext context) { }@Override public Object unmarshal(HierarchicalStreamReader reader, UnmarshallingContext context) { }@Override public boolean canConvert(Class type) { return false; } } Last one is easy as we will convert only Integer class. But there are still two methods left with these HierarchicalStreamWriter, MarshallingContext, HierarchicalStreamReader and UnmarshallingContext parameters. Luckily, we could avoid dealing with them by using AbstractSingleValueConverterthat shields us from so low level mechanisms. And now our class looks much better: public class DaysAgoConverter extends AbstractSingleValueConverter {@Override public boolean canConvert(Class type) { return type.equals(Integer.class); }@Override public Object fromString(String str) { return null; }public String toString(Object obj) { return null; } } Additionally we must override method toString(Object obj) defined in AbstractSingleValueConverter as we want to store Date in XML calculated from Integer, not a simple Object.toString value which would be returned from default toString defined in abstract parent. Implementation Code below is pretty straightforward, but most interesting lines are commented. I’ve skipped all validation stuff to make this example shorter. public class DaysAgoConverter extends AbstractSingleValueConverter {private final static String FORMAT = "yyyy-MM-dd"; // default Date format that will be used in conversion private final DateTime now = DateTime.now().toDateMidnight().toDateTime(); // current day at midnightpublic boolean canConvert(Class type) { return type.equals(Integer.class); // Converter works only with Integers }@Override public Object fromString(String str) { SimpleDateFormat format = new SimpleDateFormat(FORMAT); try { Date date = format.parse(str); return Days.daysBetween(new DateTime(date), now).getDays(); // we simply calculate days between using JodaTime } catch (ParseException e) { throw new RuntimeException("Invalid date format in " + str); } }public String toString(Object obj) { if (obj == null) { return null; }Integer daysAgo = ((Integer) obj); return now.minusDays(daysAgo).toString(FORMAT); // here we subtract days from now and return formatted date string } } Usage To use our custom converter for a specific field we have to inform about it XStream object using registerLocalConverter: xstream.registerLocalConverter(Ban.class, "daysAgo", new DaysAgoConverter()); We are using “local” method to apply this conversion only to specific field and not to every Integer field in XML file. And after that we will get our Ban objects populated with number of days instead of Date. Summary That’s all what I wanted to show you in this post. Now you have basic knowledge about what XStream is capable of and how it can be used to easily map XML data to Java objects If you need something more advanced, please check project official page as it contains very good documentation and examples.   Reference: XStream – XStreamely easy way to work with XML data in Java from our JCG partner Tomasz Dziurko at the Code Hard Go Pro blog. ...
Java Code Geeks and all content copyright © 2010-2015, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below:
Close