Featured FREE Whitepapers

What's New Here?


Under the JVM hood – Classloaders

Classloaders are a low level and often ignored aspect of the Java language among many developers. At ZeroTurnaround , our developers have had to live, breathe, eat, drink and almost get intimate with classloaders to produce the JRebel technology which interacts at a classloader level to provide live runtime class reloading, avoiding lengthy rebuilds/repackaging/redeploying cycles. Here are some of the things we’ve learnt around classloaders including some debugging tips which will hopefully save you time and potential headdesking in the future.       A classloader is just a plain java object Yes, it’s nothing clever, well other than the system classloader in the JVM, a classloader is just a java object! It’s an abstract class, ClassLoader, which can be implemented by a class you create. Here is the API: public abstract class ClassLoader {public Class loadClass(String name);protected Class defineClass(byte[] b);public URL getResource(String name);public Enumeration getResources(String name);public ClassLoader getParent()} Looks pretty straightforward, right? Let’s take a look method by method. The central method is loadClass which just takes a String class name and returns you the actual Class object. This is the method which if you’ve used classloaders before is probably the most familiar as it’s the most used in day to day coding. defineClass is a final method in the JVM that takes a byte array from a file or a location on the network and produces the same outcome, a Class object. A classloader can also find resources from a classpath. It works in a similar way to the loadClass method. There are a couple of methods, getResource and getResources, which return a URL or an Enumeration of URLs which point to the resource which represents the name passed as input to the method. Every classloader has a parent; getParent returns the classloaders parent, which is not Java inheritance related, rather a linked list style connection. We will look into this in a little more depth later on. Classloaders are lazy, so classes are only ever loaded when they are requested at runtime. Classes are loaded by the resource which invokes the class, so a class, at runtime, could be loaded by multiple classloaders depending on where they are referenced from and which classloader loaded the classes which referen… oops, I’ve gone cross-eyed! Let’s look at some code. public class A {public void doSmth() {B b = new B();b.doSmthElse();}} Here we have class A calling the constructor of class B within the doSmth of it’s methods. Under the covers this is what is happening A.class.getClassLoader().loadClass(“B”); The classloader which originally loaded class A is invoked to load the class B. Classloaders are hierarchical, but like children, they don’t always ask their parents Every classloader has a parent classloader. When a classloader is asked for a class, it will typically go straight to the parent classloader first calling loadClass which may in turn ask it’s parent and so on. If two classloaders with the same parent are asked to load the same class, it would only be done once, by the parent. It gets very troublesome when two classloaders load the same class separately, as this can cause problems which we’ll look at later. When the JEE spec was designed, the web classloader was designed to work the opposite way – great. Let’s take a look at the figure below as our example.Module WAR1 has its own classloader and prefers to load classes itself rather than delegate to it’s parent, the classloader scoped by App1.ear. This means different WAR modules, like WAR1 and WAR2 cannot see each others classes. The App1.ear module has its own classloader and is parent to the WAR1 and WAR2 classloaders. The App1.ear classloader is used by the WAR1 and WAR2 classloaders when they needs to delegate a request up the hierarchy i.e. a class is required outside of the WAR classloader scope. Effectively the WAR classes override the EAR classes where both exist. Finally the EAR classloader’s parent is the container classloader. The EAR classloader will delegate requests to the container classloader, but it does not do it in the same way as the WAR classloader, as the EAR classloader will actually prefer to delegate up rather than prefer local classes. As you can see this is getting quite hairy and is different to the plain JSE class loading behaviour. The flat classpath We talked about how the system classloader looks to the classpath to find classes that have been requested. This classpath could include directories or JAR files and the order which they are looked through is actually dependant on the JVM you are using. There may be multiple copies or versions of the class you require on the classpath, but you will always get the first instance of the class found on the classpath. It’s essentially just a list of resources, which is why it’s referred to as flat. As a result the classpath list can often be relatively slow to iterate through when looking for a resource. Problems can occur when applications who are using the same classpath want to use different versions of a class, lets use Hibernate as an example. When two versions of Hibernate JARs exist on the classpath, one version cannot be higher up the classpath for one application than it is for the other, which means both will have to use the same version. One way around this is to bloat the application (WAR) with all the libraries necessary, so that they use their local resources, but this then leads to big applications which are hard to maintain. Welcome to JAR hell! OSGi provides a solution here as it allows versioning of JAR files, or bundles, which results in a mechanism to allow wiring to particular versions of JAR files avoiding the flat classpath problems. How do I debug my class loading errors? NoClassDefFoundError/ClassNotFoundException/ClassNoDefFoundException? So, you’ve got an error/exception like the ones above. Well, does the class actually exist? Don’t bother looking in your IDE, as that’s where you compiled your class, it must be there otherwise you’ll get a compile time exception. This is a runtime exception so it’s in the runtime we want to look for the class which it says we’re missing… but where do you start? Consider the following piece of code… Arrays.toString((((URLClassLoader) Test.class.getClassLoader()) .getURLs())); This code returns an array list of all jars and directories on the classpath of the classloader the class Test is using. So now we can see if the JAR or location our mystery class should exist in is actually on the classpath. If it does not exist, add it! If it does exist, check the JAR/directory to make sure your class actually exists in that location and add it if it’s missing. These are the two typical problems which result in this error case. NoSuchMethodError/NoSuchFieldError/AbstractMethodError/IllegalAccessError? Now it’s getting interesting! These are all subclasses of the IncompatibleClassChangeError. We know the classloader has found the class we want (by name), but clearly it hasn’t found the right version. Here we have a class called Test which is making an invocation to another class, Util, but BANG – We get an exception! Lets look at the next snippet of code to debug: Test.class.getClassLoader().getResource(Util.class.getName() .replace('.', '/') + ".class"); We’re calling getResource on the classloader of class Test. This returns us the URL of the Util resource. Notice we’ve replaced the ‘.’ with a ‘/’ and added a ‘.class’ at the end of the String. This changes the package and classname of the class we’re looking for (from the perspective of the classloader) into a directory structure and filename on the filesystem – neat. This will show us the exact class we have loaded and we can make sure it’s the correct version. We can use javap -private on the class at a command prompt to see the byte code and check which methods and fields actually exist. You can easily see the structure of the class and validate whether it’s you or the Java runtime which is going crazy! Believe me, at one stage or another you’ll question both, and nearly every time it will be you! LinkageError/ClassCastException/IllegalAccessError These can occur if two different classloaders load the same class and they try to interact… ouch! Yes, it’s now getting a bit hairy. This can cause problems as we do not know if they will load the classes from the same place. How can this happen? Lets look at the following code, still in the Test class: Factory.instance().sayHello(); The code looks pretty clean and safe, and it’s not clear how an error could emerge from this line. We’re calling a static factory method to get us an instance of the Test class and are invoking a method on it. Lets look at this supporting image to show the reason why an exception is being thrown.Here we can see a web classloader (which loaded the Test class) will prefer local classes, so when it makes reference to a class, it will be loaded by the web classloader, if possible. Fairly straightforward so far. The Test class uses the Factory class to get hold of an instance of the Util class which is fairly typical practice in Java, but the Factory class doesn’t exist in the WAR as it is an external library. This is no problem as the web classloader can delegate to the shared classloader, which can see the Factory class. Note that the shared classloader is now loading it’s own version of the Util class as when the Factory instantiates the class, it uses the shared classloader (as shown in the first example earlier). The Factory class returns the Util object (created by the shared classloader) back to the WAR, which then tries to use the class, and effectively cast the class to a potentially different version of the same class (the Util class visible to the web classloader). BOOM! We can run the same code as before from within both places (The Factory.instance() method and the Test class) to see where each of our Util classes are being loaded from. Test.class.getClassLoader().getResource(Util.class.getName() .replace('.', '/') + ".class")); Hopefully this has given you an insight into the world of classloading, and instead of not understanding the classloader, you can now appreciate it with a hint of fear and uncertainty! Thanks for reading and making it to the end. We’d all like to wish you a Merry Christmas and a happy new year from ZeroTurnaround! Happy coding!   Reference: Under the JVM hood – Classloaders from our JCG partner Simon Maple at the Java Advent Calendar blog. ...

Who Do You Promote Into Management?

I vividly remember my first promotion into management. I was looking for a promotion to be a senior engineer. I asked for a promotion. I got a promotion into management. Was I ready? Oh no! I remember asking for another promotion. I was told, “You’re too valuable where you are.” I decided to make my myself less valuable and leave. When I made my last transition into management—the one where I did not transition back into development or testing—that was the one where there were two candidates for the position. One was a very technical guy who barely had any people skills and didn’t like managing people. How did I know? He said so. I was the other candidate. Now, you need to know that I have been working on my people skills my entire life. I’ve been given feedback that I’m too blunt and direct. I suspect that if I’d been born a man and 6 feet tall, I would have received kudos for being aggressive. I’m just too short and the wrong gender. On the other hand, I need to know how to phrase the information so the other people can hear it. Promoting people into management is one of those very difficult decisions. It should not be a decision you make on the spur of the moment. If you have one-on-one’s with people, you can discover their career plans. You can help them, if they want it. Part of a manager’s job is succession planning. Do you plan to be in this job forever? I hope not. Even if you just take a vacation, you are not going to be in this job for the rest of your life. You need to think about who you promote into management. Who is the best person to promote? It might not be the person with the best technical skills. It might not be the person with the best people skills. It might be the person with some combination of the two. I don’t know. You should do an analysis of the value the job requires. Here’s what I do know. If you always take the best technical person, you deprive the team of someone who was doing great technical work. And, if that person does not want to do management work, you deprive the team of a potentially great manager. If you know of someone who falls into the trap of promoting the best technical person into management, have that person read my new myth, Management Myth #12: I Must Promote the Best Technical Person to Be a Manager. Remember before, when I said I asked for a promotion? I wanted to be a manager. Why? Because I was ready for the challenge of making the difficult management decisions. I saw the project portfolio decisions that were not being made and I wanted to make them. I saw the client decisions that were not being made and I wanted to make them. I knew there were difficult tradeoffs to make in the projects, and I was willing to make them. Those were management decisions. I was willing to take a stand and make them. They were not technical decisions. They were management decisions. So, think about who you promote into management. It should not be a spur-of-the-moment decision. Think about your succession planning. Discuss what people want out of their careers in your one-on-ones with your staff. Whatever you do, don’t fall prey to the Myth: I Must Promote the Best Technical Person to Be A Manager.   Reference: Who Do You Promote Into Management? from our JCG partner Johanna Rothman at the Managing Product Development blog. ...

Chunk Oriented Processing in Spring Batch

Big Data Sets’ Processing is one of the most important problem in the software world. Spring Batch is a lightweight and robust batch framework to process the data sets. Spring Batch Framework offers ‘TaskletStep Oriented’ and ‘Chunk Oriented’ processing style. In this article, Chunk Oriented Processing Model is explained. Also, TaskletStep Oriented Processing in Spring Batch Article is definitely suggested to investigate how to develop TaskletStep Oriented Processing in Spring Batch.           Chunk Oriented Processing Feature has come with Spring Batch v2.0. It refers to reading the data one at a time, and creating ‘chunks’ that will be written out, within a transaction boundary. One item is read from an ItemReader, handed to an ItemProcessor, and written. Once the number of items read equals the commit interval, the entire chunk is written out via the ItemWriter, and then the transaction is committed. Basically, this feature should be used if at least one data item’ s reading and writing is required. Otherwise, TaskletStep Oriented processing can be used if the data item’ s only reading or writing is required. Chunk Oriented Processing model exposes three important interface as ItemReader, ItemProcessor and ItemWriter via org.springframework.batch.item package.ItemReader : This interface is used for providing the data. It reads the data which will be processed. ItemProcessor : This interface is used for item transformation. It processes input object and transforms to output object. ItemWriter : This interface is used for generic output operations. It writes the datas which are transformed by ItemProcessor. For example, the datas can be written to database, memory or outputstream (etc). In this sample application, we will write to database.Let us take a look how to develop Chunk Oriented Processing Model. Used Technologies :JDK 1.7.0_09 Spring 3.1.3 Spring Batch 2.1.9 Hibernate 4.1.8 Tomcat JDBC 7.0.27 MySQL 5.5.8 MySQL Connector 5.1.17 Maven 3.0.4STEP 1 : CREATE MAVEN PROJECT A maven project is created as below. (It can be created by using Maven or IDE Plug-in).STEP 2 : LIBRARIES A new USER Table is created by executing below script: CREATE TABLE ONLINETECHVISION.USER ( id int(11) NOT NULL AUTO_INCREMENT, name varchar(45) NOT NULL, surname varchar(45) NOT NULL, PRIMARY KEY (`id`) ); STEP 3 : LIBRARIES Firstly, dependencies are added to Maven’ s pom.xml. <properties> <spring.version>3.1.3.RELEASE</spring.version> <spring-batch.version>2.1.9.RELEASE</spring-batch.version> </properties><dependencies><dependency> <groupId>org.springframework</groupId> <artifactId>spring-core</artifactId> <version>${spring.version}</version> </dependency><dependency> <groupId>org.springframework</groupId> <artifactId>spring-context</artifactId> <version>${spring.version}</version> </dependency><dependency> <groupId>org.springframework</groupId> <artifactId>spring-tx</artifactId> <version>${spring.version}</version> </dependency><dependency> <groupId>org.springframework</groupId> <artifactId>spring-orm</artifactId> <version>${spring.version}</version> </dependency><dependency> <groupId>org.springframework.batch</groupId> <artifactId>spring-batch-core</artifactId> <version>${spring-batch.version}</version> </dependency><!-- Hibernate dependencies --> <dependency> <groupId>org.hibernate</groupId> <artifactId>hibernate-core</artifactId> <version>4.1.8.Final</version> </dependency><!-- Tomcat DBCP --> <dependency> <groupId>org.apache.tomcat</groupId> <artifactId>tomcat-jdbc</artifactId> <version>7.0.27</version> </dependency><!-- MySQL Java Connector library --> <dependency> <groupId>mysql</groupId> <artifactId>mysql-connector-java</artifactId> <version>5.1.17</version> </dependency><!-- Log4j library --> <dependency> <groupId>log4j</groupId> <artifactId>log4j</artifactId> <version>1.2.16</version> </dependency></dependencies> maven-compiler-plugin(Maven Plugin) is used to compile the project with JDK 1.7 <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.0</version> <configuration> <source>1.7</source> <target>1.7</target> </configuration> </plugin> The following Maven plugin can be used to create runnable-jar, <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.0</version><executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <configuration> <source>1.7</source> <target>1.7</target> </configuration> <transformers> <transformer implementation='org.apache.maven.plugins.shade.resource. ManifestResourceTransformer'> <mainClass>com.onlinetechvision.exe.Application</mainClass> </transformer> <transformer implementation='org.apache.maven.plugins.shade.resource. AppendingTransformer'> <resource>META-INF/spring.handlers</resource> </transformer> <transformer implementation='org.apache.maven.plugins.shade.resource. AppendingTransformer'> <resource>META-INF/spring.schemas</resource> </transformer> </transformers> </configuration> </execution> </executions> </plugin> STEP 4 : CREATE User ENTITY User Entity is created. This entity will be stored after processing. package com.onlinetechvision.user;import javax.persistence.Column; import javax.persistence.Entity; import javax.persistence.GeneratedValue; import javax.persistence.GenerationType; import javax.persistence.Id; import javax.persistence.Table;/** * User Entity * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ @Entity @Table(name='USER') public class User {private int id; private String name; private String surname;@Id @GeneratedValue(strategy=GenerationType.AUTO) @Column(name='ID', unique = true, nullable = false) public int getId() { return id; }public void setId(int id) { this.id = id; }@Column(name='NAME', unique = true, nullable = false) public String getName() { return name; }public void setName(String name) { this.name = name; }@Column(name='SURNAME', unique = true, nullable = false) public String getSurname() { return surname; }public void setSurname(String surname) { this.surname = surname; }@Override public String toString() { StringBuffer strBuff = new StringBuffer(); strBuff.append('id : ').append(getId()); strBuff.append(', name : ').append(getName()); strBuff.append(', surname : ').append(getSurname()); return strBuff.toString(); } } STEP 5 : CREATE IUserDAO INTERFACE IUserDAO Interface is created to expose data access functionality. package com.onlinetechvision.user.dao;import java.util.List;import com.onlinetechvision.user.User;/** * User DAO Interface * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ public interface IUserDAO {/** * Adds User * * @param User user */ void addUser(User user);/** * Gets User List * */ List<User> getUsers(); } STEP 6 : CREATE UserDAO IMPL UserDAO Class is created by implementing IUserDAO Interface. package com.onlinetechvision.user.dao;import java.util.List;import org.hibernate.SessionFactory;import com.onlinetechvision.user.User;/** * User DAO * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ public class UserDAO implements IUserDAO {private SessionFactory sessionFactory;/** * Gets Hibernate Session Factory * * @return SessionFactory - Hibernate Session Factory */ public SessionFactory getSessionFactory() { return sessionFactory; }/** * Sets Hibernate Session Factory * * @param SessionFactory - Hibernate Session Factory */ public void setSessionFactory(SessionFactory sessionFactory) { this.sessionFactory = sessionFactory; }/** * Adds User * * @param User user */ @Override public void addUser(User user) { getSessionFactory().getCurrentSession().save(user); }/** * Gets User List * * @return List - User list */ @SuppressWarnings({ 'unchecked' }) @Override public List<User> getUsers() { List<User> list = getSessionFactory().getCurrentSession().createQuery('from User').list(); return list; }} STEP 7 : CREATE IUserService INTERFACE IUserService Interface is created for service layer. package com.onlinetechvision.user.service;import java.util.List;import com.onlinetechvision.user.User;/** * * User Service Interface * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ public interface IUserService {/** * Adds User * * @param User user */ void addUser(User user);/** * Gets User List * * @return List - User list */ List<User> getUsers(); } STEP 8 : CREATE UserService IMPL UserService Class is created by implementing IUserService Interface. package com.onlinetechvision.user.service;import java.util.List;import org.springframework.transaction.annotation.Transactional;import com.onlinetechvision.user.User; import com.onlinetechvision.user.dao.IUserDAO;/** * * User Service * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ @Transactional(readOnly = true) public class UserService implements IUserService {IUserDAO userDAO;/** * Adds User * * @param User user */ @Transactional(readOnly = false) @Override public void addUser(User user) { getUserDAO().addUser(user); }/** * Gets User List * */ @Override public List<User> getUsers() { return getUserDAO().getUsers(); }public IUserDAO getUserDAO() { return userDAO; }public void setUserDAO(IUserDAO userDAO) { this.userDAO = userDAO; } } STEP 9 : CREATE TestReader IMPL TestReader Class is created by implementing ItemReader Interface. This class is called in order to read items. When read method returns null, reading operation is completed. The following steps explains with details how to be executed firstJob. The commit-interval value of firstjob is 2 and the following steps are executed : 1) firstTestReader is called to read first item(firstname_0, firstsurname_0) 2) firstTestReader is called again to read second item(firstname_1, firstsurname_1) 3) testProcessor is called to process first item(FIRSTNAME_0, FIRSTSURNAME_0) 4) testProcessor is called to process second item(FIRSTNAME_1, FIRSTSURNAME_1) 5) testWriter is called to write first item(FIRSTNAME_0, FIRSTSURNAME_0) to database 6) testWriter is called to write second item(FIRSTNAME_1, FIRSTSURNAME_1) to database 7) first and second items are committed and the transaction is closed. firstTestReader is called to read third item(firstname_2, firstsurname_2) 9) maxIndex value of firstTestReader is 3. read method returns null and item reading operation is completed. 10) testProcessor is called to process third item(FIRSTNAME_2, FIRSTSURNAME_2) 11) testWriter is called to write first item(FIRSTNAME_2, FIRSTSURNAME_2) to database 12) third item is committed and the transaction is closed. firstStep is completed with COMPLETED status and secondStep is started. secondJob and thirdJob are executed in the same way. package com.onlinetechvision.item;import org.springframework.batch.item.ItemReader; import org.springframework.batch.item.NonTransientResourceException; import org.springframework.batch.item.ParseException; import org.springframework.batch.item.UnexpectedInputException;import com.onlinetechvision.user.User;/** * TestReader Class is created to read items which will be processed * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ public class TestReader implements ItemReader<User> { private int index; private int maxIndex; private String namePrefix; private String surnamePrefix;/** * Reads items one by one * * @return User * * @throws Exception * @throws UnexpectedInputException * @throws ParseException * @throws NonTransientResourceException * */ @Override public User read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException { User user = new User(); user.setName(getNamePrefix() + '_' + index); user.setSurname(getSurnamePrefix() + '_' + index);if(index > getMaxIndex()) { return null; }incrementIndex();return user; }/** * Increments index which defines read-count * * @return int * */ private int incrementIndex() { return index++; }public int getMaxIndex() { return maxIndex; }public void setMaxIndex(int maxIndex) { this.maxIndex = maxIndex; }public String getNamePrefix() { return namePrefix; }public void setNamePrefix(String namePrefix) { this.namePrefix = namePrefix; }public String getSurnamePrefix() { return surnamePrefix; }public void setSurnamePrefix(String surnamePrefix) { this.surnamePrefix = surnamePrefix; }} STEP 10 : CREATE FailedCaseTestReader IMPL FailedCaseTestReader Class is created in order to simulate the failed job status. In this sample application, when thirdJob is processed at fifthStep, failedCaseTestReader is called and exception is thrown so its status will be FAILED. package com.onlinetechvision.item;import org.springframework.batch.item.ItemReader; import org.springframework.batch.item.NonTransientResourceException; import org.springframework.batch.item.ParseException; import org.springframework.batch.item.UnexpectedInputException;import com.onlinetechvision.user.User;/** * FailedCaseTestReader Class is created in order to simulate the failed job status. * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ public class FailedCaseTestReader implements ItemReader<User> { private int index; private int maxIndex; private String namePrefix; private String surnamePrefix;/** * Reads items one by one * * @return User * * @throws Exception * @throws UnexpectedInputException * @throws ParseException * @throws NonTransientResourceException * */ @Override public User read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException { User user = new User(); user.setName(getNamePrefix() + '_' + index); user.setSurname(getSurnamePrefix() + '_' + index);if(index >= getMaxIndex()) { throw new Exception('Unexpected Error!'); }incrementIndex();return user; }/** * Increments index which defines read-count * * @return int * */ private int incrementIndex() { return index++; }public int getMaxIndex() { return maxIndex; }public void setMaxIndex(int maxIndex) { this.maxIndex = maxIndex; }public String getNamePrefix() { return namePrefix; }public void setNamePrefix(String namePrefix) { this.namePrefix = namePrefix; }public String getSurnamePrefix() { return surnamePrefix; }public void setSurnamePrefix(String surnamePrefix) { this.surnamePrefix = surnamePrefix; }} STEP 11 : CREATE TestProcessor IMPL TestProcessor Class is created by implementing ItemProcessor Interface. This class is called to process items. User item is received from TestReader, processed and returned to TestWriter. package com.onlinetechvision.item;import java.util.Locale;import org.springframework.batch.item.ItemProcessor;import com.onlinetechvision.user.User;/** * TestProcessor Class is created to process items. * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ public class TestProcessor implements ItemProcessor<User, User> {/** * Processes items one by one * * @param User user * @return User * @throws Exception * */ @Override public User process(User user) throws Exception { user.setName(user.getName().toUpperCase(Locale.ENGLISH)); user.setSurname(user.getSurname().toUpperCase(Locale.ENGLISH)); return user; }} STEP 12 : CREATE TestWriter IMPL TestWriter Class is created by implementing ItemWriter Interface. This class is called to write items to DB, memory etc… package com.onlinetechvision.item;import java.util.List;import org.springframework.batch.item.ItemWriter;import com.onlinetechvision.user.User; import com.onlinetechvision.user.service.IUserService;/** * TestWriter Class is created to write items to DB, memory etc... * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ public class TestWriter implements ItemWriter<User> {private IUserService userService;/** * Writes items via list * * @throws Exception * */ @Override public void write(List<? extends User> userList) throws Exception { for(User user : userList) { getUserService().addUser(user); } System.out.println('User List : ' + getUserService().getUsers()); }public IUserService getUserService() { return userService; }public void setUserService(IUserService userService) { this.userService = userService; }} STEP 13 : CREATE FailedStepTasklet CLASS FailedStepTasklet is created by implementing Tasklet Interface. It illustrates business logic in failed step. package com.onlinetechvision.tasklet;import org.apache.log4j.Logger; import org.springframework.batch.core.StepContribution; import org.springframework.batch.core.scope.context.ChunkContext; import org.springframework.batch.core.step.tasklet.Tasklet; import org.springframework.batch.repeat.RepeatStatus;/** * FailedStepTasklet Class illustrates a failed job. * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ public class FailedStepTasklet implements Tasklet {private static final Logger logger = Logger.getLogger(FailedStepTasklet.class);private String taskResult;/** * Executes FailedStepTasklet * * @param StepContribution stepContribution * @param ChunkContext chunkContext * @return RepeatStatus * @throws Exception * */ public RepeatStatus execute(StepContribution stepContribution, ChunkContext chunkContext) throws Exception { logger.debug('Task Result : ' + getTaskResult()); throw new Exception('Error occurred!'); }public String getTaskResult() { return taskResult; }public void setTaskResult(String taskResult) { this.taskResult = taskResult; }} STEP 14 : CREATE BatchProcessStarter CLASS BatchProcessStarter Class is created to launch the jobs. Also, it logs their execution results. package com.onlinetechvision.spring.batch;import org.apache.log4j.Logger; import org.springframework.batch.core.Job; import org.springframework.batch.core.JobExecution; import org.springframework.batch.core.JobParametersBuilder; import org.springframework.batch.core.JobParametersInvalidException; import org.springframework.batch.core.launch.JobLauncher; import org.springframework.batch.core.repository.JobExecutionAlreadyRunningException; import org.springframework.batch.core.repository.JobInstanceAlreadyCompleteException; import org.springframework.batch.core.repository.JobRepository; import org.springframework.batch.core.repository.JobRestartException;/** * BatchProcessStarter Class launches the jobs and logs their execution results. * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ public class BatchProcessStarter {private static final Logger logger = Logger.getLogger(BatchProcessStarter.class);private Job firstJob; private Job secondJob; private Job thirdJob; private JobLauncher jobLauncher; private JobRepository jobRepository;/** * Starts the jobs and logs their execution results. * */ public void start() { JobExecution jobExecution = null; JobParametersBuilder builder = new JobParametersBuilder();try { getJobLauncher().run(getFirstJob(), builder.toJobParameters()); jobExecution = getJobRepository().getLastJobExecution(getFirstJob().getName(), builder.toJobParameters()); logger.debug(jobExecution.toString());getJobLauncher().run(getSecondJob(), builder.toJobParameters()); jobExecution = getJobRepository().getLastJobExecution(getSecondJob().getName(), builder.toJobParameters()); logger.debug(jobExecution.toString());getJobLauncher().run(getThirdJob(), builder.toJobParameters()); jobExecution = getJobRepository().getLastJobExecution(getThirdJob().getName(), builder.toJobParameters()); logger.debug(jobExecution.toString());} catch (JobExecutionAlreadyRunningException | JobRestartException | JobInstanceAlreadyCompleteException | JobParametersInvalidException e) { logger.error(e); }}public Job getFirstJob() { return firstJob; }public void setFirstJob(Job firstJob) { this.firstJob = firstJob; }public Job getSecondJob() { return secondJob; }public void setSecondJob(Job secondJob) { this.secondJob = secondJob; }public Job getThirdJob() { return thirdJob; }public void setThirdJob(Job thirdJob) { this.thirdJob = thirdJob; }public JobLauncher getJobLauncher() { return jobLauncher; }public void setJobLauncher(JobLauncher jobLauncher) { this.jobLauncher = jobLauncher; }public JobRepository getJobRepository() { return jobRepository; }public void setJobRepository(JobRepository jobRepository) { this.jobRepository = jobRepository; }} STEP 15 : CREATE dataContext.xml jdbc.properties, is created. It defines data-source informations and is read via dataContext.xml jdbc.db.driverClassName=com.mysql.jdbc.Driver jdbc.db.url=jdbc:mysql://localhost:3306/onlinetechvision jdbc.db.username=root jdbc.db.password=root jdbc.db.initialSize=10 jdbc.db.minIdle=3 jdbc.db.maxIdle=10 jdbc.db.maxActive=10 jdbc.db.testWhileIdle=true jdbc.db.testOnBorrow=true jdbc.db.testOnReturn=true jdbc.db.initSQL=SELECT 1 FROM DUAL jdbc.db.validationQuery=SELECT 1 FROM DUAL jdbc.db.timeBetweenEvictionRunsMillis=30000 STEP 16 : CREATE dataContext.xml Spring Configuration file, dataContext.xml, is created. It covers dataSource, sessionFactory and transactionManager definitions. <?xml version='1.0' encoding='UTF-8'?> <beans xmlns='http://www.springframework.org/schema/beans' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:context='http://www.springframework.org/schema/context' xmlns:p='http://www.springframework.org/schema/p' xmlns:batch='http://www.springframework.org/schema/batch' xmlns:tx='http://www.springframework.org/schema/tx' xsi:schemaLocation='http://www.springframework.org/schema/beanshttp://www.springframework.org/schema/beans/spring-beans-3.0.xsdhttp://www.springframework.org/schema/contexthttp://www.springframework.org/schema/context/spring-context-3.0.xsdhttp://www.springframework.org/schema/batchhttp://www.springframework.org/schema/batch/spring-batch-2.1.xsdhttp://www.springframework.org/schema/txhttp://www.springframework.org/schema/tx/spring-tx-3.0.xsd'><context:property-placeholder location='classpath:jdbc.properties'/><!-- Enable the configuration of transactional behavior based on annotations --> <tx:annotation-driven transaction-manager='transactionManager'/><!-- Data Source Declaration --> <bean id='dataSource' class='org.apache.tomcat.jdbc.pool.DataSource' destroy-method='close' p:driverClassName='${jdbc.db.driverClassName}' p:url='${jdbc.db.url}' p:username='${jdbc.db.username}' p:password='${jdbc.db.password}' p:initialSize='${jdbc.db.initialSize}' p:minIdle='${jdbc.db.minIdle}' p:maxIdle='${jdbc.db.maxIdle}' p:maxActive='${jdbc.db.maxActive}' p:testWhileIdle='${jdbc.db.testWhileIdle}' p:testOnBorrow='${jdbc.db.testOnBorrow}' p:testOnReturn='${jdbc.db.testOnReturn}' p:initSQL='${jdbc.db.initSQL}' p:validationQuery='${jdbc.db.validationQuery}' p:timeBetweenEvictionRunsMillis='${jdbc.db.timeBetweenEvictionRunsMillis}'/><!-- Session Factory Declaration --> <bean id='sessionFactory' class='org.springframework.orm.hibernate4.LocalSessionFactoryBean'> <property name='dataSource' ref='dataSource' /> <property name='annotatedClasses'> <list> <value>com.onlinetechvision.user.User</value> </list> </property> <property name='hibernateProperties'> <props> <prop key='hibernate.dialect'>org.hibernate.dialect.MySQLDialect</prop> <prop key='hibernate.show_sql'>true</prop> </props> </property> </bean><!-- Transaction Manager Declaration --> <bean id='transactionManager' class='org.springframework.orm.hibernate4.HibernateTransactionManager'> <property name='sessionFactory' ref='sessionFactory'/> </bean></beans> STEP 17 : CREATE jobContext.xml Spring Configuration file, jobContext.xml, is created. It covers jobRepository, jobLauncher, item reader, item processor, item writer, tasklet and job definitions. <?xml version='1.0' encoding='UTF-8'?> <beans xmlns='http://www.springframework.org/schema/beans' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:batch='http://www.springframework.org/schema/batch' xsi:schemaLocation='http://www.springframework.org/schema/beanshttp://www.springframework.org/schema/beans/spring-beans-3.0.xsdhttp://www.springframework.org/schema/batchhttp://www.springframework.org/schema/batch/spring-batch-2.1.xsd'><import resource='dataContext.xml'/><!-- jobRepository Declaration --> <bean id='jobRepository' class='org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean'> <property name='transactionManager' ref='transactionManager' /> </bean><!-- jobLauncher Declaration --> <bean id='jobLauncher' class='org.springframework.batch.core.launch.support.SimpleJobLauncher' > <property name='jobRepository' ref='jobRepository'/> </bean><!-- Reader Bean Declarations --> <bean id='firstTestReader' class='com.onlinetechvision.item.TestReader'> <property name='maxIndex' value='2'/> <property name='namePrefix' value='firstname'/> <property name='surnamePrefix' value='firstsurname'/> </bean><bean id='secondTestReader' class='com.onlinetechvision.item.TestReader'> <property name='maxIndex' value='2'/> <property name='namePrefix' value='secondname'/> <property name='surnamePrefix' value='secondsurname'/> </bean><bean id='thirdTestReader' class='com.onlinetechvision.item.TestReader'> <property name='maxIndex' value='3'/> <property name='namePrefix' value='thirdname'/> <property name='surnamePrefix' value='thirdsurname'/> </bean><bean id='fourthTestReader' class='com.onlinetechvision.item.TestReader'> <property name='maxIndex' value='3'/> <property name='namePrefix' value='fourthname'/> <property name='surnamePrefix' value='fourthsurname'/> </bean><bean id='fifthTestReader' class='com.onlinetechvision.item.TestReader'> <property name='maxIndex' value='3'/> <property name='namePrefix' value='fifthname'/> <property name='surnamePrefix' value='fifthsurname'/> </bean><bean id='failedCaseTestReader' class='com.onlinetechvision.item.FailedCaseTestReader'> <property name='maxIndex' value='1'/> <property name='namePrefix' value='failedcasename'/> <property name='surnamePrefix' value='failedcasesurname'/> </bean><!-- Processor Bean Declaration --> <bean id='testProcessor' class='com.onlinetechvision.item.TestProcessor' /><!-- Writer Bean Declaration --> <bean id='testWriter' class='com.onlinetechvision.item.TestWriter' > <property name='userService' ref='userService'/> </bean><!-- Failed Step Tasklet Declaration --> <bean id='failedStepTasklet' class='com.onlinetechvision.tasklet.FailedStepTasklet'> <property name='taskResult' value='Error occurred!' /> </bean><!-- Batch Job Declarations --> <batch:job id='firstJob'> <batch:step id='firstStep' next='secondStep'> <batch:tasklet> <batch:chunk reader='firstTestReader' processor='testProcessor' writer='testWriter' commit-interval='2'/> </batch:tasklet> </batch:step> <batch:step id='secondStep'> <batch:tasklet> <batch:chunk reader='secondTestReader' processor='testProcessor' writer='testWriter' commit-interval='2'/> </batch:tasklet> </batch:step> </batch:job><batch:job id='secondJob'> <batch:step id='thirdStep'> <batch:tasklet> <batch:chunk reader='thirdTestReader' processor='testProcessor' writer='testWriter' commit-interval='2'/> </batch:tasklet> <batch:next on='*' to='fourthStep' /> <batch:next on='FAILED' to='firstFailedStep' /> </batch:step> <batch:step id='fourthStep'> <batch:tasklet> <batch:chunk reader='fourthTestReader' processor='testProcessor' writer='testWriter' commit-interval='2'/> </batch:tasklet> </batch:step> <batch:step id='firstFailedStep'> <batch:tasklet ref='failedStepTasklet' /> </batch:step> </batch:job><batch:job id='thirdJob'> <batch:step id='fifthStep'> <batch:tasklet> <batch:chunk reader='failedCaseTestReader' processor='testProcessor' writer='testWriter' commit-interval='2'/> </batch:tasklet> <batch:next on='*' to='sixthStep' /> <batch:next on='FAILED' to='secondFailedStep' /> </batch:step> <batch:step id='sixthStep'> <batch:tasklet> <batch:chunk reader='fifthTestReader' processor='testProcessor' writer='testWriter' commit-interval='2'/> </batch:tasklet> </batch:step> <batch:step id='secondFailedStep'> <batch:tasklet ref='failedStepTasklet' /> </batch:step> </batch:job></beans> STEP 18 : CREATE applicationContext.xml Spring Configuration file, applicationContext.xml, is created. It covers bean definitions. <?xml version='1.0' encoding='UTF-8'?> <beans xmlns='http://www.springframework.org/schema/beans' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xmlns:batch='http://www.springframework.org/schema/batch' xsi:schemaLocation='http://www.springframework.org/schema/beanshttp://www.springframework.org/schema/beans/spring-beans-3.0.xsdhttp://www.springframework.org/schema/batchhttp://www.springframework.org/schema/batch/spring-batch-2.1.xsd'><import resource='jobContext.xml'/><!-- User DAO Declaration --> <bean id='userDAO' class='com.onlinetechvision.user.dao.UserDAO'> <property name='sessionFactory' ref='sessionFactory' /> </bean><!-- User Service Declaration --> <bean id='userService' class='com.onlinetechvision.user.service.UserService'> <property name='userDAO' ref='userDAO' /> </bean><!-- BatchProcessStarter Declaration --> <bean id='batchProcessStarter' class='com.onlinetechvision.spring.batch.BatchProcessStarter'> <property name='jobLauncher' ref='jobLauncher'/> <property name='jobRepository' ref='jobRepository'/> <property name='firstJob' ref='firstJob'/> <property name='secondJob' ref='secondJob'/> <property name='thirdJob' ref='thirdJob'/> </bean></beans> STEP 19 : CREATE Application CLASS Application Class is created to run the application. package com.onlinetechvision.exe;import org.springframework.context.ApplicationContext; import org.springframework.context.support.ClassPathXmlApplicationContext;import com.onlinetechvision.spring.batch.BatchProcessStarter;/** * Application Class starts the application. * * @author onlinetechvision.com * @since 10 Dec 2012 * @version 1.0.0 * */ public class Application {/** * Starts the application * * @param String[] args * */ public static void main(String[] args) { ApplicationContext appContext = new ClassPathXmlApplicationContext('applicationContext.xml'); BatchProcessStarter batchProcessStarter = (BatchProcessStarter)appContext.getBean('batchProcessStarter'); batchProcessStarter.start(); }} STEP 20 : BUILD PROJECT After OTV_SpringBatch_Chunk_Oriented_Processing Project is built, OTV_SpringBatch_Chunk_Oriented_Processing-0.0.1-SNAPSHOT.jar will be created. STEP 21 : RUN PROJECT After created OTV_SpringBatch_Chunk_Oriented_Processing-0.0.1-SNAPSHOT.jar file is run, the following database and console output logs will be shown : Database screenshot :First Job’ s console output : 16.12.2012 19:30:41 INFO (SimpleJobLauncher.java:118) - Job: [FlowJob: [name=firstJob]] launched with the following parameters: [{}]16.12.2012 19:30:41 DEBUG (AbstractJob.java:278) - Job execution starting: JobExecution: id=0, version=0, startTime=null, endTime=null, lastUpdated=Sun Dec 16 19:30:41 GMT 2012, status=STARTING, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=0, version=0, JobParameters=[{}], Job=[firstJob]]User List : [id : 181, name : FIRSTNAME_0, surname : FIRSTSURNAME_0, id : 182, name : FIRSTNAME_1, surname : FIRSTSURNAME_1, id : 183, name : FIRSTNAME_2, surname : FIRSTSURNAME_2, id : 184, name : SECONDNAME_0, surname : SECONDSURNAME_0, id : 185, name : SECONDNAME_1, surname : SECONDSURNAME_1, id : 186, name : SECONDNAME_2, surname : SECONDSURNAME_2]16.12.2012 19:30:42 DEBUG (BatchProcessStarter.java:43) - JobExecution: id=0, version=2, startTime=Sun Dec 16 19:30:41 GMT 2012, endTime=Sun Dec 16 19:30:42 GMT 2012, lastUpdated=Sun Dec 16 19:30:42 GMT 2012, status=COMPLETED, exitStatus=exitCode=COMPLETED;exitDescription=, job=[JobInstance: id=0, version=0, JobParameters=[{}], Job=[firstJob]] Second Job’ s console output : 16.12.2012 19:30:42 INFO (SimpleJobLauncher.java:118) - Job: [FlowJob: [name=secondJob]] launched with the following parameters: [{}]16.12.2012 19:30:42 DEBUG (AbstractJob.java:278) - Job execution starting: JobExecution: id=1, version=0, startTime=null, endTime=null, lastUpdated=Sun Dec 16 19:30:42 GMT 2012, status=STARTING, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=1, version=0, JobParameters=[{}], Job=[secondJob]]User List : [id : 181, name : FIRSTNAME_0, surname : FIRSTSURNAME_0, id : 182, name : FIRSTNAME_1, surname : FIRSTSURNAME_1, id : 183, name : FIRSTNAME_2, surname : FIRSTSURNAME_2, id : 184, name : SECONDNAME_0, surname : SECONDSURNAME_0, id : 185, name : SECONDNAME_1, surname : SECONDSURNAME_1, id : 186, name : SECONDNAME_2, surname : SECONDSURNAME_2, id : 187, name : THIRDNAME_0, surname : THIRDSURNAME_0, id : 188, name : THIRDNAME_1, surname : THIRDSURNAME_1, id : 189, name : THIRDNAME_2, surname : THIRDSURNAME_2, id : 190, name : THIRDNAME_3, surname : THIRDSURNAME_3, id : 191, name : FOURTHNAME_0, surname : FOURTHSURNAME_0, id : 192, name : FOURTHNAME_1, surname : FOURTHSURNAME_1, id : 193, name : FOURTHNAME_2, surname : FOURTHSURNAME_2, id : 194, name : FOURTHNAME_3, surname : FOURTHSURNAME_3]16.12.2012 19:30:42 DEBUG (BatchProcessStarter.java:47) - JobExecution: id=1, version=2, startTime=Sun Dec 16 19:30:42 GMT 2012, endTime=Sun Dec 16 19:30:42 GMT 2012, lastUpdated=Sun Dec 16 19:30:42 GMT 2012, status=COMPLETED, exitStatus=exitCode=COMPLETED;exitDescription=, job=[JobInstance: id=1, version=0, JobParameters=[{}], Job=[secondJob]] Third Job’ s console output : 16.12.2012 19:30:42 INFO (SimpleJobLauncher.java:118) - Job: [FlowJob: [name=thirdJob]] launched with the following parameters: [{}]16.12.2012 19:30:42 DEBUG (AbstractJob.java:278) - Job execution starting: JobExecution: id=2, version=0, startTime=null, endTime=null, lastUpdated=Sun Dec 16 19:30:42 GMT 2012, status=STARTING, exitStatus=exitCode=UNKNOWN;exitDescription=, job=[JobInstance: id=2, version=0, JobParameters=[{}], Job=[thirdJob]]16.12.2012 19:30:42 DEBUG (TransactionTemplate.java:159) - Initiating transaction rollback on application exception org.springframework.batch.repeat.RepeatException: Exception in batch process; nested exception is java.lang.Exception: Unexpected Error! ...16.12.2012 19:30:43 DEBUG (BatchProcessStarter.java:51) - JobExecution: id=2, version=2, startTime=Sun Dec 16 19:30:42 GMT 2012, endTime=Sun Dec 16 19:30:43 GMT 2012, lastUpdated=Sun Dec 16 19:30:43 GMT 2012, status=FAILED, exitStatus=exitCode=FAILED;exitDescription=, job=[JobInstance: id=2, version=0, JobParameters=[{}], Job=[thirdJob]] STEP 22 : DOWNLOAD https://github.com/erenavsarogullari/OTV_SpringBatch_Chunk_Oriented_Processing Resources: Chunk Oriented Processing in Spring Batch   Reference: Chunk Oriented Processing in Spring Batch from our JCG partner Eren Avsarogullari at the Online Technology Vision blog. ...

Java Object resurrection

Overview After an object which overrides finalize() is collected it is added to a finalization queue to be cleaned up after calling the finalize() method of each object.  By what happens if you resurrect the object?             When is finalize called? The finalize method is called by a single threaded system task which calls this method for each object which has been collected. Note: the nodes in the finalization queue are objects also have finalize() methods notionally. Objects cannot be cleaned up until the GC after they have been finalized. Most objects (including the node in the finalization queue) don’t overriden finalize() and so the GC is smart enough to detect this and not add them to the queue. These obejcts can be cleaned up immediately. If you override the method, even with an empty one, it makes a difference. What about resurrected objects? In the finalize() method, you can resurrect the object by giving making something point to it. e.g. a static collection.  This object can no longer be collected by a GC (until it is discarded again)  So what happens then? The object has been flags as being finalized once and is not finalized repeatedly. static final List ZOMBIES = new ArrayList<>(); static class Zombies { private int num; public Zombies(int num) { this.num = num; } @Override protected void finalize() throws Throwable { System.out.println("Resurrect " + num); ZOMBIES.add(this); } @Override public String toString() { return "Zombies{" + "num=" + num + '}'; } } public static void main(String... args) throws InterruptedException { for (int i = 0; i < 3; i++) ZOMBIES.add(new Zombies(i)); for (int j = 0; j < 5; j++) { System.out.println("Zombies: " + ZOMBIES); ZOMBIES.clear(); System.gc(); Thread.sleep(100); } } printsZombies: [Zombies{num=0}, Zombies{num=1}, Zombies{num=2}] Resurrect 2 Resurrect 1 Resurrect 0 Zombies: [Zombies{num=2}, Zombies{num=1}, Zombies{num=0}] Zombies: [] Zombies: [] Zombies: []In this example, the Zombies are added once to the collection and resurrected once by the finalize method. When they are collected a second time, they have been flagged as finalized and not queued again. Conclusion While it’s a good idea to avoid using finalize(), it is a small comfort to know it will only be called once it the object is resurrected.   Reference: Java Object resurrection from our JCG partner Peter Lawrey at the Vanilla Java blog. ...

Don’t take the Technical Debt Metaphor too far

Because “technical debt” has the word “debt” in it, many people have decided that it makes sense to think and work with technical debt in monetary terms, and treat technical debt as a real financial cost. This is supposed to make it easier for technical people to explain technical debt to the business, and easier to make a business case for paying debt off. Putting technical debt into financial terms also allows consultants and vendors to try to scare business executives into buying their tools or their help – like Gartner calculating that world wide “IT debt” costs will exceed $1.5 in a couple of more years, or CAST software’s assessment that the average enterprise is carrying millions of dollars of technical debt. Businesses understand debt. Businesses make a decision to take on debt on and they track it, account for it and manage it. The business always knows how much debt they have, why they took it on, and when they need to pay it off. Businesses don’t accidentally take on debt – debt doesn’t just show up on the books one day. We don’t know when we’re taking technical debt on But developers accidentally take on debt all of the time – what Martin Fowler calls “inadvertent debt”, due to inexperience and misunderstandings, everything from “What’s Layering?” to “Now we know how we should have done it” looking at the design a year or two later.‘The point is that while you’re programming, you are learning. It’s often the case that it can take a year of programming on a project before you understand what the best design approach should have been.’ Taking on this kind of debt is inevitable – and you’ll never know when you’re taking it on or how much, because you don’t know what you don’t know. Even when developers take on debt consciously, they don’t understand the costs at the time – the principal or the interest. Most teams don’t record when they make a trade-off in design or a shortcut in coding or test automation, never mind try to put a value on paying off their choice. We don’t understand (or often even see) technical debt costs until long after we’ve taken the costs on. When you’re dealing with quality and stability problems; or when you’re estimating out a change and you recognize that you made mistakes in the past or that you took shortcuts that you didn’t realize before or shortcuts that you did know about but that turned out to be much more expensive than you expected; or once you understand that you chose the wrong architecture or the wrong technical platform. Or maybe you’ve just run a static analysis tool like CAST or SONAR which tells you that you have thousands of dollars of technical debt in your code base that you didn’t know about until now. Now try and explain to a business executive that you just realized or just remembered that you have put the company into debt for tens or hundreds of thousands of dollars. Businesses don’t and can’t run this way. We don’t know how much technical debt is really costing us By expressing everything in financial terms, we’re also pretending that technical debt costs are all hard costs to the business and that we actually know how much the principal and interest costs are: we’re $100,000 in debt and the interest rate is 3% per year. Assigning a monetary value to technical debt costs give them a false sense of precision and accuracy. Let’s be honest. There aren’t clear and consistent criteria for costing technical debt and modelling technical debt repayment – we don’t even have a definition of what technical debt is that we can all agree on. Two people can come up with a different technical debt assessment for the same system, because what I think technical debt is and what you think technical debt is aren’t the same. And just because a tool says that technical debt costs are $100,000.00 for a code base, doesn’t make the number true. Any principal and interest that you calculate (or some tool calculates for you) are made-up numbers and the business will know this when you try to defend them – which you are going to have to do, if you want to talk in financial terms with someone who does finance for a living. You’re going to be on shaky ground at best – at worse, they’ll understand that you’re not talking about real business debt and wonder what you’re trying to pull off. The other problem that I see is “debt fatigue”. Everyone is overwhelmed by the global government debt crisis and the real estate debt crisis and the consumer debt crisis and the fiscal cliff and whatever comes next. Your business may be already fighting its own problems with managing its financial debt. Technical debt is one more argument about debt that nobody is looking forward to hearing. We don’t need to talk about debt with the business We don’t use the term “technical debt” with the business, or try to explain it in financial debt terms. If we need to rewrite code because it is unstable, we treat this like any other problem that needs to be solved – we cost it out, explain the risks, and prioritize this work with everything else. If we need to rewrite or restructure code in order to make upcoming changes easier, cheaper and less risky, we explain this as part of the work that needs to be done, and justify the costs. If we need to replace or upgrade a platform technology because we are getting poor support from the supplier, we consider this a business risk that needs to be understood and managed. And if code should be refactored or tests filled in, we don’t explain it, we just do it as part of day-to-day engineering work. We’re dealing with technical debt in terms that the business understands without using a phony financial model. We’re not pretending that we’re carrying off-balance sheet debt that the company needs to rely on technologists to understand and manage. We’re leaving debt valuation and payment amortization arguments to the experts in finance and accounting where they belong, and focusing on solving problems in software, which is where we belong.   Reference: Don’t take the Technical Debt Metaphor too far from our JCG partner Jim Bird at the Building Real Software blog. ...

Java – far sight look at JDK 8

The world is changing slowly but surely. After the changes that gave java a fresher look with JDK 7, the java community is looking forward to the rest of the improvements that will come with JDK 8 and probably JDK 9. The targeted purpose of JDK 8 was to fill in the gaps in the implementation of JDK 7 – part of the remaining puzzle pieces laking from this implementation, that should be available for the broad audience by in late 2013 is to improve and boost the language in three particular directions:productivity performance modularitySo from next year, java will run everywhere (mobile, cloud, desktop, server etc.), but in an improved manner. In what follows I will provide a short overview of what to expect from 2013 – just in time for New Year’s Resolutions – afterwards I will focus mainly on productivity side with emphasis on project lambda and how will its introduction affect the way we code. Productivity In regards of productivity JDK 8 targets two main areas: – collections – a more facile way to interact with collections through literal extensions brought to the language – annotations – enhanced support for annotations, allowing writting them in contexts where are currently illegal (e.g. primitives) Performance The addition of the Fork/Join framework to JDK 7, was the first step that java took in the direction of multicore CPUs. JDK 8 takes this road even further by bringing closures’ support to java (lambda expression, that is). Probably the most affected part of java will be the Collections part, the closures combined with the newly added interfaces and functionalities pushing the java containers to the next level. Besides the more readable and shorter code to be written, by providing to the collections a lambda expression that will be executed internally the platform can take advantage of multicore processors. Modularity One of the most interresting pieces for the community was project jigsaw: ‘The goal of this Project is to design and implement a standard module system for the Java SE Platform, and to apply that system to the Platform itself and to the JDK.’. I am using past tense because, for the those of us that were hoping to get rid of the classpaths and classloaders, we have to postpone our exciment for Java 9, as for that point of time was also project jigsaw postponed. To have a clearer picture of how the remaning Java Roadmap 2013:2013/01/31 M6 Feature Complete 2013/02/21 M7 Developer Preview 2013/07/05 M8 Final Release Candidate 2013/09/09 GA General AvailabilityBesides project jigsaw another big and exciting change that will come (in this version), is the support for closures. Provided through the help of lambda expressions they will improve key points of the JDK. Lambdas Getting started First and first of all one should get a lambda enabled SDK. In this direction there are two ways to obtain one: * the one intended for the brave ones: build it from the sources * the convenient one: downloading an already compiled version of the SDK Initially I started with building it from the sources, but due to the lack of time and too many warnings related to environment variables, I opted for the lazy approach and took the already existing JDK. The other important tool is a text editor to write the code. As it happened until now, tipically first came the JDK release and after a period of time, an enabled IDE came out. This time it is different, maybe also due to the transparency and the broad availability of the SDK through openjdk. Some days ago, the first Java 8 enabled IDE was realesed by JetBrain. So IntelliJ IDEA version 12 is the first IDE to provide support for JDK 8, besides are improvements? So for testing purposes I used IntelliJ 12 Community Edition together with JDK 8 b68, on a Windows 7, x64 machine. For those of you that prefer Netbeans, a nightly build with lambda support is available for download. Adjusting to the appropriate mindset. Before starting to write improved and cleaner code using the newly provided features, one must get a grasp on a couple new concepts – I needed to, anyway.What is a lambda expression? The easiest way to see a lambda expression is just like a method: ‘it provides a list of formal parameters and a body—an expression or block— expressed in terms of those parameters.The parameters of a lambda expression can be either declared or inferred, when the formal parameters have inferred types, then these types are derived from the functional interface type targeted by the lambda expression. From the point of view of the returned value, a lambda expression can be void compatible – they don’t return anything or value compatible – if any given execution path returns a value. Examples of lambda expressions:(a) (int a, int b) -> a + b(b) (int a, int b) -> { if (a > b) { return a; } else if (a == b) { return a * b; } else { return b; } }What is a functional interface? A functional interface is an interface that contains just one abstract method, hence represents a single method contract. In some situations, the single method may have the form of multiple methods with override-equivalent signatures, in this case all the methods represent a single method. Besides the typical way of creating an interface instance by creating and instantiating a class, functional interface instances can be created also by usage of lambda expressions, method or constructor references. Example of functional interfaces: // custom built functional interface public interface FuncInterface { public void invoke(String s1, String s2); } Example of functional interfaces from the JAVA API: java.lang.Comparable java.lang.Runnable java.util.concurrent.Callable java.awt.event.ActionListener So let’s see how the starting of a thread might change in the future: OLD WAY: new Thread(new Runnable() { @Override public void run() { for (int i=0; i< 9; i++) { System.out.println(String.format('Message #%d from inside the thread!', i)); } } }).start(); NEW WAY: new Thread(() -> { for (int i=0; i< 9; i++) { System.out.println(String.format('Message #%d from inside the thread!', i)); } }).start(); Even if I didn’t write for some time any java Swing, AWT related functionality I have to admit that lambdas will give a breath of fresh air to the Swing developers Action listener addition: JButton button = new JButton('Click');// NEW WAY: button.addActionListener( (e) -> { System.out.println('The button was clicked!'); });// OLD WAY: button.addActionListener(new ActionListener() { @Override public void actionPerformed(ActionEvent e) { System.out.println('The button was clicked using old fashion code!'); } });Who/What is SAM? SAM stands for Single Abstract Method, so to cut some corners we can say that SAM == functional interface. Even if in the initial specification, also abstract classes with only one abstract method were considered SAM types, some people found/guessed also the reason why. Method/Constructor referencingThe lambdas sound all nice and all? But somehow the need for functional interface is somehow to some extend restrictive – does this mean that I can use only interfaces that contain a single abstract method? Not really – JDK 8 provides an aliasing mechanism that allows ‘extraction’ of methods from classes or objects. This can be done by using the newly added :: operator. It can be applied on classes – for extraction of static methods or on objects for extraction of methods. The same operator can be used for constructors also. Referencing: interface ConstructorReference{ T constructor(); }interface MethodReference { void anotherMethod(String input); }public class ConstructorClass { String value;public ConstructorClass() { value = 'default'; }public static void method(String input) { System.out.println(input); }public void nextMethod(String input) { // operations }public static void main(String... args) { // constructor reference ConstructorReferencereference = ConstructorClass::new; ConstructorClass cc = reference.constructor();// static method reference MethodReference mr = cc::method;// object method reference MethodReference mr2 = cc::nextMethod;System.out.println(cc.value); } }Default methods in interfacesThis means that from version 8, java interfaces can contain method bodies, so to put it simple java will support multiple inheritance without the headaches that usually come with it. Also, by providing default implementations for interface methods one can assure ensure that adding a new method will not create chaos in the implementing classes. JDK 8 added default methods to interfaces like java.util.Collection or java.util.Iterator and through this it provided a mechanism to better use lambdas where it is really needed. Notable interfaces added: java.util.stream.Streamable java.util.stream.Stream Improved collections’ interaction In my opinion all the changes that come with project lambda are great additions to the language, that will make it align with the current day standards and make it simpler and leaner but probably the change that will have the biggest productivity impact and the biggest cool + wow effect is definitely the revamping of the collections framework. No, there is no Collection 2 framework, we still have to cope with type erasure for now, but java will make another important shift: from external to internal iteration. By doing so, it provides the developer the mechanism to filter and aggregate collections in an elegant manner and besides this to push for more efficiency. By providing a lambda expression that will be executed internally, so multicore processors can be used to their full power. Let’s consider the following scenarios: a. Considering a list of strings, select all of them that are uppercased written. How would this be written? OLD WAY: //..... List inputList = new LinkedList<>(); List upper = new LinkedList<>();// add elementsfor (String currentValue : inputList) { if (currentValue != null && currentValue.matches("[A-Z0-9]*")) { upper.add(currentValue); } }System.out.println(upper); //….. NEW WAY: //..... inputList.stream().filter(x -> (x != null && x.matches('[A-Z0-9]*'))).into(upper); b. Consider that you would like to change all the extracted characters to lowercase. Using the JDK8 way this would look like this: // ..... inputList.stream().filter(x -> (x != null && x.matches("[A-Z0-9]*"))).map(String::toLowerCase).into(upper); c. And how about finding out the number of characters from the selected collection // ..... int sumX = inputList.stream().filter(x -> (x != null && x.matches("[A-Z0-9]*"))).map(String::length).reduce(0, Integer::sum); Used methods: default Streamstream() // java.util.Collection Streamfilter(Predicate predicate) // java.util.stream.Stream IntStream map(IntFunction mapper) //java.util.stream.Stream d. What if I would like to take each element from a collection and print it? //OLD WAY: for (String current : list) { System.out.println(current); } //NEW WAY: list.forEach(x -> System.out.println(x)); Besides the mentioned functionality, JDK 8 has are other interesting news also, but for brevity reasons I will stop here. More information about it can be found on the JDK 8 Project lambda site or the webpage of the JSR 337. To conclude, Java is moving forward and I personally like the direction it is heading, another point of interest would be to point of time when library developers start adopting JDK 8 too. That will be for sure interesting. Thank you for your time and patience, I wish you a merry Christmas. Resources Brian Goetz resource folder: http://cr.openjdk.java.net/~briangoetz/lambda Method/constructor references: http://doanduyhai.wordpress.com/2012/07/14/java-8-lambda-in-details-part-iii-method-and-constructor-referencing   Reference: Java – far sight look at JDK 8 from our JCG partner Olimpiu Pop at the Java Advent Calendar blog. ...

Generating Barcodes in PDFs with Flying-Saucer

Flying-Saucer is a nice library to generate PDF documents from within Java applications. Just generate a bunch of XHTML, throw it into the renderer and let it produce the desired document utilizing iText. When it comes to barcodes however, Flying-Saucer cannot access the built in barcode functionality of iText (at least I didn’t find any documentation for it).             However, being OpenSource and well designed, one only needs to create one subclass to achieve the task: Flying-Saucer relies on a factory named ReplacedElementFactory, which can replace elements by custom objects. This is also used to embed images, as the class ITextReplacedElementFactory shows. Now we can simply create a subclass, which replaces images with an appropriate barcode: <img src=’0123456789′ type=’code128′ style=’height: 1cm’ /> One simply needs to override the createReplacedElement method like this (the whole code can be found here: BarcodeReplacedElementFactory.java (GitHub)): @Override public ReplacedElement createReplacedElement( LayoutContext c, BlockBox box, UserAgentCallback uac, int cssWidth, int cssHeight) {Element e = box.getElement(); if (e == null) { return null; }String nodeName = e.getNodeName(); if (nodeName.equals("img")) { if ("code128".equals(e.getAttribute("type"))) { try { Barcode128 code = new Barcode128(); code.setCode(e.getAttribute("src")); FSImage fsImage = new ITextFSImage( Image.getInstance( code.createAwtImage( Color.BLACK, Color.WHITE ), Color.WHITE )); if (cssWidth != -1 || cssHeight != -1) { fsImage.scale(cssWidth, cssHeight); } return new ITextImageElement(fsImage); } catch (Throwable e1) { return null; } } }return super.createReplacedElement( c, box, uac, cssWidth, cssHeight); } Granted, ‘type’ is no valid XHTML-Element for <img /> but as you can see in the code above, you could easily replace it with data-type or any other attribute. Flying-Saucer doesn’t seem to care about this anyway. Note: The code above can only handle Code128-Barcodes, but can easily be extended to handle EAN and the like (iText supports a whole bunch of barcodes by default). In order to make our factory work, we need to pass it to the renderer – which is pretty darn easy: ITextRenderer renderer = new ITextRenderer(); renderer.getSharedContext().setReplacedElementFactory( new BarcodeReplacedElementFactory( renderer.getOutputDevice() )); renderer.setDocumentFromString(inputAsString); renderer.layout(); renderer.createPDF(outputAsStream);   Reference: Generating Barcodes in PDFs with Flying-Saucer from our JCG partner Andreas Haufler at the Andy’s Software Engineering Corner blog. ...

Hand and Finger Detection using JavaCV

This post is part of a series on Natural User Interfaces (NUIs) published by Dr. Andrew Davison and deals with detecting hands from a webcam video feed using JavaCV. Note: all the source code for this chapter is available for download from http://fivedots.coe.psu.ac.th/~ad/jg/nui055/             The colored blob detection code of chapter 5 (available at http://fivedots.coe.psu.ac.th/~ad/jg/nui05/) can be used as the basis of other shape analyzers, which I’ll illustrate here by extending it to detect a hand and fingers. In Figure 1, I’m wearing a black glove on my left hand. My Handy application attempts to find and label the thumb, index, middle, ring, and little finger. Yellow lines are drawn between the fingertips and the center-of-gravity (COG) of the hand.I utilized the HSVSelector application of chapter 5 to determine suitable HSV ranges for the black glove. These ranges are loaded by Handy prior to executing the steps shown in Figure 2 to obtain the hand’s contour, its COG, and orientation relative to the horizontal.The various stages in Figure 2 are almost identical to those carried out by the ColorRectDetector.findRect() method in section 4.1 of chapter 5. However, Handy continues processing, employing a convex hull and convexity defects to locate and label the fingertips within the hand contour. These additional steps are shown in Figure 3.The hull and defects are obtained from the contour with standard OpenCV operations, which I’ll explain below. However, the final step of naming the fingers utilizes a rather hacky strategy that assumes the contour’s defects are for an out-stretched left hand. The thumb and index finger are located based on their angular position relative to the COG, and the other fingers are identified based on their position relative to those fingers. This process is rather fragile, and can easily become confused, as shown in Figure 4.Nevertheless, the technique is fairly reliable, usually identifying at least the thumb and index finger irrespective of the hand’s orientation, which should be enough for basic gesture processing. However, the application doesn’t identify gestures, which will hopefully be the subject of a later chapter. Class diagrams for Handy are shown in Figure 5, with only the public methods listed.The top-levels of Handy parallel those of the BlobsDrumming application in chapter 5 (e.g. see chapter 5’s Figure 11), with the Handy class managing the JFrame and HandPanel displaying the annotated webcam image. The image analyses summarized in Figures 2 and 3 are performed by the HandDetector class, which is passed the current webcam snap via a call to update(). It draws the current labeled fingertips, COG, and connecting lines when HandPanel calls HandDetector.draw(). 1. Analyzing the Webcam Image The update() method is essentially a series of calls that implement Figures 2 and 3. // globals private static final int IMG_SCALE = 2; // scaling applied to webcam image// HSV ranges defining the glove color private int hueLower, hueUpper, satLower, satUpper, briLower, briUpper;// OpenCV elements private IplImage hsvImg; // HSV version of webcam image private IplImage imgThreshed; // threshold for HSV settings// hand details private Point cogPt; // center of gravity (COG) of contour private int contourAxisAngle; // contour's main axis angle relative to the horiz (in degrees) private ArrayList fingerTips;public void update(BufferedImage im) { BufferedImage scaleIm = scaleImage(im, IMG_SCALE); // reduce the size of the image to make processing faster// convert image format to HSV cvCvtColor(IplImage.createFrom(scaleIm), hsvImg, CV_BGR2HSV);// threshold image using loaded HSV settings for user's glove cvInRangeS(hsvImg, cvScalar(hueLower, satLower, briLower, 0), cvScalar(hueUpper, satUpper, briUpper, 0), imgThreshed);cvMorphologyEx(imgThreshed, imgThreshed, null, null, CV_MOP_OPEN, 1); // erosion followed by dilation on the image to remove // specks of white while retaining the image sizeCvSeq bigContour = findBiggestContour(imgThreshed); if (bigContour == null) return;extractContourInfo(bigContour, IMG_SCALE); // find the COG and angle to horizontal of the contourfindFingerTips(bigContour, IMG_SCALE); // detect the fingertips positions in the contournameFingers(cogPt, contourAxisAngle, fingerTips); } // end of update() update() begins by scaling the supplied webcam image to improve processing speeds. It then converts the picture into HSV format so it can generate a threshold image using the black glove’s HSV ranges. This corresponds to the first line of Figure 2, although the threshold is actually rendered as white pixels against a black background. The threshold, minus small specks, is passed to findBiggestContour(); the resulting contour is assumed to be the user’s hand in subsequent processing stages. extractContourInfo() analyzes the contour to find the hand’s center-of-gravity (COG), and its orientation relative to the horizontal, which are stored in the cogPt and contourAxisAngle globals. The completion of extractContourInfo() corresponds to the end of Figure 2. The findFingerTips() method wraps a convex hull around the contour to identify the shape’s defects (the top line of Figure 3), which we assume are the hand’s fingers. After a little filtering to reduce the number of defects, the remaining ones are treated as fingertip coordinates, and stored in the global fingerTips list. nameFingers() labels the fingers (assuming that the thumb and index finger are on the left side of the hand), completing Figure 3’s stages. 1.1 Finding the Biggest Contour findBiggestContour() uses the OpenCV function cvFindContours() to create a list of contours. For my binary threshold image, a contour is a region (or blob) of white pixels. Each blob is approximated by a bounding box, and the contour corresponding to the largest box is selected and returned. // globals private static final float SMALLEST_AREA = 600.0f; // ignore smaller contour areasprivate CvMemStorage contourStorage;private CvSeq findBiggestContour(IplImage imgThreshed) { CvSeq bigContour = null; // generate all the contours in the threshold image as a list CvSeq contours = new CvSeq(null); cvFindContours(imgThreshed, contourStorage, contours, Loader.sizeof(CvContour.class), CV_RETR_LIST, CV_CHAIN_APPROX_SIMPLE);// find the largest contour in the list based on bounded box size float maxArea = SMALLEST_AREA; CvBox2D maxBox = null; while (contours != null && !contours.isNull()) { if (contours.elem_size() > 0) { CvBox2D box = cvMinAreaRect2(contours, contourStorage); if (box != null) { CvSize2D32f size = box.size(); float area = size.width() * size.height(); if (area > maxArea) { maxArea = area; bigContour = contours; } } } contours = contours.h_next(); } return bigContour; } // end of findBiggestContour() cvFindContours() can return different types of contours, collected together in different kinds of data structures. I generate the simplest kind of contours, storing them in a linear list which can be searched with a while loop. After some experimentation, I placed a lower bound on the bounded box size of 600 square pixels which filters out small boxes surrounding patches of image noise. This means that findBiggestContour() may return null if it doesn’t find a large enough box. 1.2 Calculating the COG and Horizontal Angle The next step shown in Figure 2 is to find the COG and angle to the horizontal of the hand contour by calling extractContourInfo(). This is where the code in HandDetector starts to part company from the analysis carried out by ColorRectDetector.findRect() in chapter 5. In section 4.2 of that chapter, the enclosing box around the contour is utilized to obtain a center and orientation. This is adequate because the underlying shape is a rectangular card, and so the contour and box are almost identical. However, a bounding box around a hand can easily have a very different COG or angle from the hand itself; in this case, it’s necessary to directly analyze the hand contour rather than a bounding box, by utilizing moments. I used spatial moments back in chapter 3 to find the COG of a binary image. The same technique can be applied to a contour to find its center (or centroid). I can also calculate second order mixed moments, which give information about the spread of pixels around the centroid. Second order moments can be combined to return the orientation (or angle) of the contour’s major axis relative to the x-axis. Referring back to the OpenCV moments notation from chapter 3, the m() moments function is defined as:The function takes two arguments, p and q, which are used as powers for x and y. The I() function is the intensity for a pixel defined by its (x, y) coordinate. n is the number of pixels that make up the shape. If I consider a contour like the one in Figure 6, then θ is the angle of its major axis to the horizontal, with the +y-axis pointing downwards.In terms of the m() function, it can be shown that:The extractContourInfo() method shown below uses spatial moments to obtain the contour’s centroid, and cvGetCentralMoment() to calculate the major axis angle according to the above equation; these results are stored in the globals cogPt and contourAxisAngle for use later. // globals private Point cogPt; // center of gravity (COG) of contour private int contourAxisAngle; // contour's main axis angle relative to horizontal (in degrees)private ArrayList fingerTips;private void extractContourInfo(CvSeq bigContour, int scale) { CvMoments moments = new CvMoments(); cvMoments(bigContour, moments, 1);// center of gravity double m00 = cvGetSpatialMoment(moments, 0, 0) ; double m10 = cvGetSpatialMoment(moments, 1, 0) ; double m01 = cvGetSpatialMoment(moments, 0, 1);if (m00 != 0) { // calculate center int xCenter = (int) Math.round(m10/m00)*scale; int yCenter = (int) Math.round(m01/m00)*scale; cogPt.setLocation(xCenter, yCenter); }double m11 = cvGetCentralMoment(moments, 1, 1); double m20 = cvGetCentralMoment(moments, 2, 0); double m02 = cvGetCentralMoment(moments, 0, 2); contourAxisAngle = calculateTilt(m11, m20, m02);// deal with hand contour pointing downwards /* uses fingertips information generated on the last update of the hand, so will be out-of-date */if (fingerTips.size() > 0) { int yTotal = 0; for(Point pt : fingerTips) yTotal += pt.y; int avgYFinger = yTotal/fingerTips.size(); if (avgYFinger > cogPt.y) // fingers below COG contourAxisAngle += 180; } contourAxisAngle = 180 - contourAxisAngle; /* this makes the angle relative to a positive y-axis that runs up the screen */ } // end of extractContourInfo()private int calculateTilt(double m11, double m20, double m02) { double diff = m20 - m02; if (diff == 0) { if (m11 == 0) return 0; else if (m11 > 0) return 45; else // m11 < 0 return -45; }double theta = 0.5 * Math.atan2(2*m11, diff); int tilt = (int) Math.round( Math.toDegrees(theta));if ((diff > 0) && (m11 == 0)) return 0; else if ((diff < 0) && (m11 == 0)) return -90; else if ((diff > 0) && (m11 > 0)) // 0 to 45 degrees return tilt; else if ((diff > 0) && (m11 < 0)) // -45 to 0 return (180 + tilt); // change to counter-clockwise angle else if ((diff < 0) && (m11 > 0)) // 45 to 90 return tilt; else if ((diff < 0) && (m11 < 0)) // -90 to -45 return (180 + tilt); // change to counter-clockwise angleSystem.out.println("Error in moments for tilt angle"); return 0; } // end of calculateTilt() Moments in OpenCV are explained in depth in ‘Simple Image Analysis by Moments’ by Johannes Kilian at http://public.cranfield.ac.uk/c5354/teaching/dip/opencv/SimpleImageAnalysisbyMoments.pdf. The code inside calculateTilt() is based on the special cases for θ listed in Table 1 of Kilian’s paper. Unfortunately, the axis angle doesn’t distinguish between a hand with fingers pointing upwards and one facing down, and so it’s necessary to examine the relative position of the fingertips with respect to the COG to decide whether the angle should be adjusted. The problem is that this information isn’t available until after the hand contour’s convex hull has been examined for defects, which occurs after extractContourInfo() has finished. My solution is to use the fingertip coordinates calculated in the previous call to update() which analyzed the webcam frame before the current one. The data will be out-of-date, but the hand won’t have moved much in the 200 ms interval between snaps. 1.3 Finding the Fingertips Identifying the fingertips is carried out in the first row of Figure 3; in the code, a convex hull is wrapped around the contour by OpenCV’s cvConvexHull2() and this polygon is compared to the contour by cvConvexityDefects() to find its defects. Hull creation and defect analysis are speeded up by utilizing a low-polygon approximation of the contour rather than the original. These stages are performed in the first half of the findFingerTips() method: // globals private static final int MAX_POINTS = 20; // max number of points stored in an array// OpenCV elements private CvMemStorage contourStorage, approxStorage, hullStorage, defectsStorage;// defects data for the hand contour private Point[] tipPts, foldPts; private float[] depths;private void findFingerTips(CvSeq bigContour, int scale) { CvSeq approxContour = cvApproxPoly(bigContour, Loader.sizeof(CvContour.class), approxStorage, CV_POLY_APPROX_DP, 3, 1); // reduce number of points in the contourCvSeq hullSeq = cvConvexHull2(approxContour, hullStorage, CV_COUNTER_CLOCKWISE, 0); // find the convex hull around the contourCvSeq defects = cvConvexityDefects(approxContour, hullSeq, defectsStorage); // find the defect differences between the contour and hullint defectsTotal = defects.total(); if (defectsTotal > MAX_POINTS) { System.out.println("Processing " + MAX_POINTS + " defect pts"); defectsTotal = MAX_POINTS; }// copy defect information from defects sequence into arrays for (int i = 0; i < defectsTotal; i++) { Pointer pntr = cvGetSeqElem(defects, i); CvConvexityDefect cdf = new CvConvexityDefect(pntr);CvPoint startPt = cdf.start(); tipPts[i] = new Point( (int)Math.round(startPt.x()*scale), (int)Math.round(startPt.y()*scale)); // array contains coords of the fingertipsCvPoint endPt = cdf.end(); CvPoint depthPt = cdf.depth_point(); foldPts[i] = new Point( (int)Math.round(depthPt.x()*scale), (int)Math.round(depthPt.y()*scale)); //array contains coords of the skin fold between fingersdepths[i] = cdf.depth()*scale; // array contains distances from tips to folds }reduceTips(defectsTotal, tipPts, foldPts, depths); } // end of findFingerTips() The latter half of findFingerTips() extracts the tip and fold coordinates and depths from the defects sequence. The earlier call to the convex hull method, cvConvexHull2(), with a CV_COUNTER_CLOCKWISE argument means that the coordinates will be stored in a counter-clockwise order, like that of Figure 7.The fingertips are stored in a tipPts[] array, the finger folds (the indentations between the fingers) in foldPts[], and their depths in depths[]. As Figure 7 suggests, the analysis often generates too many defects, and so reduceTips() is called at the end of findFingerTips(). It applies two simple tests to filter out defects that are unlikely to be fingertips – it discards points with shallow defect depths, and coordinates with too great an angle between their neighboring fold points. Examples of both are shown in Figure 8.reduceTips() stores the remaining tip points in the global fingerTips list: // globals private static final int MIN_FINGER_DEPTH = 20; private static final int MAX_FINGER_ANGLE = 60; // degreesprivate ArrayList fingerTips;private void reduceTips(int numPoints, Point[] tipPts, Point[] foldPts, float[] depths) { fingerTips.clear();for (int i=0; i < numPoints; i++) { if (depths[i] < MIN_FINGER_DEPTH) // defect too shallow continue;// look at fold points on either side of a tip int pdx = (i == 0) ? (numPoints-1) : (i - 1); // predecessor of i int sdx = (i == numPoints-1) ? 0 : (i + 1); // successor of iint angle = angleBetween(tipPts[i], foldPts[pdx], foldPts[sdx]); if (angle >= MAX_FINGER_ANGLE) continue; // angle between finger and folds too wide// this point is probably a fingertip, so add to list fingerTips.add(tipPts[i]); } } // end of reduceTips()private int angleBetween(Point tip, Point next, Point prev) // calculate the angle between the tip and its neighboring folds // (in integer degrees) { return Math.abs( (int)Math.round( Math.toDegrees( Math.atan2(next.x - tip.x, next.y - tip.y) - Math.atan2(prev.x - tip.x, prev.y - tip.y)) )); } 1.4 Naming the Fingers nameFingers() uses the list of fingertip coordinates, and the contour’s COG and axis angle to label the fingers in two steps. First, it calls labelThumbIndex() to label the thumb and index finger based on their likely angles relative to the COG, assuming that they are on the left side of the hand. nameFingers() attempts to label the other fingers in labelUnknowns(), based on their known order relative to the thumb and index finger. // globals private ArrayList namedFingers;private void nameFingers(Point cogPt, int contourAxisAngle, ArrayList fingerTips) { // reset all named fingers to unknown namedFingers.clear(); for (int i=0; i < fingerTips.size(); i++) namedFingers.add(FingerName.UNKNOWN);labelThumbIndex(fingerTips, namedFingers); labelUnknowns(namedFingers); } // end of nameFingers() The Finger IDs and their relative order are maintained in a FingerName enumeration: public enum FingerName { LITTLE, RING, MIDDLE, INDEX, THUMB, UNKNOWN;public FingerName getNext() { int nextIdx = ordinal()+1; if (nextIdx == (values().length)) nextIdx = 0; return values()[nextIdx]; } // end of getNext()public FingerName getPrev() { int prevIdx = ordinal()-1; if (prevIdx < 0) prevIdx = values().length-1; return values()[prevIdx]; } // end of getPrev()} // end of FingerName enum One of the possible finger names is UNKNOWN, which is used to label all the fingertips prior to the calls to the naming methods. labelThumbIndex() attempts to label the thumb and index fingers based on the angle ranges illustrated in Figure 9.The index finger can turn between 60 and 120 degrees around the COG, while the thumb can move between 120 and 200 degrees. I arrived at these angles through trial-and-error, and they assume that the hand is orientated straight up. labelThumbIndex() also assumes that the thumb and index fingers will most likely be stored at the end of the fingerTips list, since the contour hull was built in a counter-clockwise order. It therefore increases its chances of matching against the right defects by iterating backwards through the list. // globals private static final int MIN_THUMB = 120; // angle ranges private static final int MAX_THUMB = 200;private static final int MIN_INDEX = 60; private static final int MAX_INDEX = 120;// hand details private Point cogPt private int contourAxisAngle;private void labelThumbIndex(ArrayList fingerTips, ArrayList nms) { boolean foundThumb = false; boolean foundIndex = false; int i = fingerTips.size()-1; while ((i >= 0)) { int angle = angleToCOG(fingerTips.get(i), cogPt, contourAxisAngle); // check for thumb if ((angle <= MAX_THUMB) && (angle>MIN_THUMB) && !foundThumb) { nms.set(i, FingerName.THUMB); foundThumb = true; }// check for index if ((angle <= MAX_INDEX) && (angle > MIN_INDEX) && !foundIndex) { nms.set(i, FingerName.INDEX); foundIndex = true; } i--; } } // end of labelThumbIndex() angleToCOG() calculates the angle of a fingertip relative to the COG, remembering to factor in the contour axis angle so that the hand is orientated straight up. private int angleToCOG(Point tipPt, Point cogPt, int contourAxisAngle) { int yOffset = cogPt.y - tipPt.y; // make y positive up screen int xOffset = tipPt.x - cogPt.x; double theta = Math.atan2(yOffset, xOffset); int angleTip = (int) Math.round( Math.toDegrees(theta)); return angleTip + (90 - contourAxisAngle); // this ensures that the hand is orientated straight up } // end of angleToCOG() labelUnknowns() is passed a list of finger names which hopefully contains THUMB and INDEX at certain positions and UNKNOWNs everywhere else. Using a named finger as a starting point, the UNKNOWNs are changed to finger names based on their ordering in the FingerName enumeration. private void labelUnknowns(ArrayList nms) { // find first named finger int i = 0; while ((i < nms.size()) && (nms.get(i) == FingerName.UNKNOWN)) i++; if (i == nms.size()) // no named fingers found, so give up return;FingerName name = nms.get(i); labelPrev(nms, i, name); // fill-in backwards labelFwd(nms, i, name); // fill-in forwards } // end of labelUnknowns() labelPrev() and labelFwd() differ only in the direction they move through the list of names. labelPrev() moves backwards trying to change UNKNOWNS to named fingers, but only if the name hasn’t already been assigned to the list. 2. Drawing the Named Fingers The analysis performed by update() will result in a list of fingertip points (in the fingerTips global), an associated list of named fingers (in namedFingers), and a contour COG and axis angle. All of these, apart from the angle, are utilized by draw() to add named finger labels to the webcam image, as shown in Figure 1 and Figure 10 below.An unknown finger ‘tip’ (labeled as UNKNOWN in namedFingers) is drawn as a red circle. // globals private Point cogPt; private ArrayList fingerTips; private ArrayList namedFingers;public void draw(Graphics2D g2d) { if (fingerTips.size() == 0) return;g2d.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON); // line smoothing g2d.setPaint(Color.YELLOW); g2d.setStroke(new BasicStroke(4)); // thick yellow pen// label tips in red or green, and draw lines to named tips g2d.setFont(msgFont); for (int i=0; i < fingerTips.size(); i++) { Point pt = fingerTips.get(i); if (namedFingers.get(i) == FingerName.UNKNOWN) { g2d.setPaint(Color.RED); // unnamed fingertip is red g2d.drawOval(pt.x-8, pt.y-8, 16, 16); g2d.drawString("" + i, pt.x, pt.y-10); // label with a digit } else { // draw yellow line to the named fingertip from COG g2d.setPaint(Color.YELLOW); g2d.drawLine(cogPt.x, cogPt.y, pt.x, pt.y);g2d.setPaint(Color.GREEN); // named fingertip is green g2d.drawOval(pt.x-8, pt.y-8, 16, 16); g2d.drawString(namedFingers.get(i).toString().toLowerCase(), pt.x, pt.y-10); } }// draw COG g2d.setPaint(Color.GREEN); g2d.fillOval(cogPt.x-8, cogPt.y-8, 16, 16); } // end of draw() 3. Gesture Detection The Handy application stops short at converting the named fingertips into gestures, which would require an analysis of how the fingers move through space over time. Preliminary tests show that Handy can only identify gestures reliably when they involve an outstretched thumb and/or index finger, perhaps combined with other fingers. Gestures of this type include ‘victory’, ‘wave’, ‘good’, ‘point’, and ‘gun’ shown in Figure 11.A common gesture that cannot be detected by Handy is ‘ok’ (see Figure 12) because it requires the fingers be brought together, which cannot be detected solely in terms of contour defects.  Reference: Hand and Finger Detection using JavaCV from our JCG partner Attila-Mihaly Balazs at the Java Advent Calendar blog. ...

Multi-threading in Java Swing with SwingWorker

If you’re writing a desktop or Java Web Start program in Java using Swing, you might feel the need to run some stuff in the background by creating your own threads. There’s nothing stopping you from using standard multi-threading techniques in Swing, and the usual considerations apply. If you have multiple threads accessing the same variables, you’ll need to use synchronized methods or code blocks (or thread-safe classes like AtomicInteger or ArrayBlockingQueue).           However, there is a pitfall for the unwary. As with most user interface APIs, you can’t update the user interface from threads you’ve created yourself. Well, as every Java undergraduate knows, you often can, but you shouldn’t. If you do this, sometimes your program will work and other times it won’t. You can get around this problem by using the specialised SwingWorker class. In this article, I’ll show you how you can get your programs working even if you’re using the Thread class, and then we’ll go on to look at the SwingWorker solution. For demonstration purposes, I’ve created a little Swing program.As you can see, it consists of two labels and a start button. At the moment, clicking the start button invokes a handler method which does nothing. Here’s the Java code: import java.awt.Font; import java.awt.GridBagConstraints; import java.awt.GridBagLayout; import java.awt.event.ActionEvent; import java.awt.event.ActionListener; import java.util.List; import java.util.concurrent.ExecutionException;import javax.swing.JButton; import javax.swing.JFrame; import javax.swing.JLabel; import javax.swing.SwingUtilities; import javax.swing.SwingWorker;public class MainFrame extends JFrame {private JLabel countLabel1 = new JLabel('0'); private JLabel statusLabel = new JLabel('Task not completed.'); private JButton startButton = new JButton('Start');public MainFrame(String title) { super(title);setLayout(new GridBagLayout());countLabel1.setFont(new Font('serif', Font.BOLD, 28));GridBagConstraints gc = new GridBagConstraints();gc.fill = GridBagConstraints.NONE;gc.gridx = 0; gc.gridy = 0; gc.weightx = 1; gc.weighty = 1; add(countLabel1, gc);gc.gridx = 0; gc.gridy = 1; gc.weightx = 1; gc.weighty = 1; add(statusLabel, gc);gc.gridx = 0; gc.gridy = 2; gc.weightx = 1; gc.weighty = 1; add(startButton, gc);startButton.addActionListener(new ActionListener() { public void actionPerformed(ActionEvent arg0) { start(); } });setSize(200, 400); setDefaultCloseOperation(EXIT_ON_CLOSE); setVisible(true); }private void start() {}public static void main(String[] args) { SwingUtilities.invokeLater(new Runnable() {@Override public void run() { new MainFrame('SwingWorker Demo'); } }); } } We’re going to add some code into the start() method which is called in response to the start button being clicked. First let’s try a normal thread. private void start() { Thread worker = new Thread() { public void run() {// Simulate doing something useful. for(int i=0; i<=10; i++) { // Bad practice countLabel1.setText(Integer.toString(i));try { Thread.sleep(1000); } catch (InterruptedException e) {} }// Bad practice statusLabel.setText('Completed.'); } };worker.start(); } As a matter of fact, this code seems to work (at least for me anyway). The program ends up looking like this:This isn’t recommended practice, however. We’re updating the GUI from our own thread, and under some circumstances that will certainly cause exceptions to be thrown. If we want to update the GUI from another thread, we should use SwingUtilities to schedule our update code to run on the event dispatch thread. The following code is fine, but ugly as the devil himself. private void start() { Thread worker = new Thread() { public void run() {// Simulate doing something useful. for(int i=0; i<=10; i++) {final int count = i;SwingUtilities.invokeLater(new Runnable() { public void run() { countLabel1.setText(Integer.toString(count)); } });try { Thread.sleep(1000); } catch (InterruptedException e) {} }SwingUtilities.invokeLater(new Runnable() { public void run() { statusLabel.setText('Completed.'); } });} };worker.start(); } Surely there must be something we can do to make our code more elegant? The SwingWorker Class SwingWorker is an alternative to using the Thread class, specifically designed for Swing. It’s an abstract class and it takes two template parameters, which make it look highly ferocious and puts most people off using it. But in fact it’s not as complex as it seems. Let’s take a look at some code that just runs a background thread. For this first example, we won’t be using either of the template parameters, so we’ll set them both to Void, Java’s class equivalent of the primitive void type (with a lower-case ‘v’). Running a Background Task We can run a task in the background by implementing the doInBackground method and calling execute to run our code. SwingWorker<Void, Void> worker = new SwingWorker<Void, Void>() { @Override protected Void doInBackground() throws Exception { // Simulate doing something useful. for (int i = 0; i <= 10; i++) { Thread.sleep(1000); System.out.println('Running ' + i); }return null; } };worker.execute(); Note that SwingWorker is a one-shot affair, so if we want to run the code again, we’d need to create another SwingWorker; you can’t restart the same one. Pretty simple, hey? But what if we want to update the GUI with some kind of status after running our code? You cannot update the GUI from doInBackground, because it’s not running in the main event dispatch thread. But there is a solution. We need to make use of the first template parameter. Updating the GUI After the Thread Completes We can update the GUI by returning a value from doInBackground() and then over-riding done(), which can safely update the GUI. We use the get() method to retrieve the value returned from doInBackground() So the first template parameter determines the return type of both doInBackground() and get(). SwingWorker<Boolean, Void> worker = new SwingWorker<Boolean, Void>() { @Override protected Boolean doInBackground() throws Exception { // Simulate doing something useful. for (int i = 0; i <= 10; i++) { Thread.sleep(1000); System.out.println('Running ' + i); }// Here we can return some object of whatever type // we specified for the first template parameter. // (in this case we're auto-boxing 'true'). return true; }// Can safely update the GUI from this method. protected void done() {boolean status; try { // Retrieve the return value of doInBackground. status = get(); statusLabel.setText('Completed with status: ' + status); } catch (InterruptedException e) { // This is thrown if the thread's interrupted. } catch (ExecutionException e) { // This is thrown if we throw an exception // from doInBackground. } }};worker.execute();What if we want to update the GUI as we’re going along? That’s what the second template parameter is for. Updating the GUI from a Running Thread To update the GUI from a running thread, we use the second template parameter. We call the publish() method to ‘publish’ the values with which we want to update the user interface (which can be of whatever type the second template parameter specifies). Then we override the process() method, which receives the values that we publish. Actually process() receives lists of published values, because several values may get published before process() is actually called. In this example we just publish the latest value to the user interface. SwingWorker<Boolean, Integer> worker = new SwingWorker<Boolean, Integer>() { @Override protected Boolean doInBackground() throws Exception { // Simulate doing something useful. for (int i = 0; i <= 10; i++) { Thread.sleep(1000);// The type we pass to publish() is determined // by the second template parameter. publish(i); }// Here we can return some object of whatever type // we specified for the first template parameter. // (in this case we're auto-boxing 'true'). return true; }// Can safely update the GUI from this method. protected void done() {boolean status; try { // Retrieve the return value of doInBackground. status = get(); statusLabel.setText('Completed with status: ' + status); } catch (InterruptedException e) { // This is thrown if the thread's interrupted. } catch (ExecutionException e) { // This is thrown if we throw an exception // from doInBackground. } }@Override // Can safely update the GUI from this method. protected void process(List<Integer> chunks) { // Here we receive the values that we publish(). // They may come grouped in chunks. int mostRecentValue = chunks.get(chunks.size()-1);countLabel1.setText(Integer.toString(mostRecentValue)); }};worker.execute();More …. ? You Want More …. ? I hope you enjoyed this introduction to the highly-useful SwingWorker class. You can find more tutorials, including a complete free video course on multi-threading and courses on Swing, Android and Servlets, on my site Cave of Programming.   Reference: Multi-threading in Java Swing with SwingWorker from our JCG partner John Purcell at the Java Advent Calendar blog. ...

Performance of inlined virtual method invocations in Java

Overview One of the benefits of dynamic compilation it the ability to support extensive method inlining on virtual method code. While inlining the code improves performance, the code still has to check the type (in case it has changed since it was optimised) or select between multiple possible implementations. This leads to the question; how does having multiple implementations of a method, called via an interface,  impact performance. Benchmarking This benchmark tests a trivial method invoked as elements of a list.  It compares different numbers of classes in the list to see how it performs, but also varies the amount the loop is unrolled so that the number of possible classes is reduced. e.g. for a list of 12 different classes, in a repeating pattern, with out unrolling the method invocation could be to any of the 12 classes, but when the loop is unrolled to a degree of two, each call has only 6 possible classes (12/2). When unrolled to a degree of three, there is 4 possible classes for each call (12/3), for a loop unroll to a degree of six, there is only 2 possible classes. Note: the possible classes is different for each line of code. These are the results on an 2.5 GHz i5. Times are in nanoseconds. The axis headings refer to the number of different classes used or called from a given line of code.1 used 2 used 3 used 4 used 6 used 8 used 12 used1 per call site 48.4 46.6 46.9 43.7 48.7 54.92 per call site115.880.5 92.8 87 1123 per call site2852832714 per call site669281 2756 per call site5622758 per call site49812 per call site530It appears that the number of classes loaded, or even the number of classes in the List is not as important as the number for possible classes called from a given piece of code.  It also appears that different pieces of code in the same loop can be optimised independantly. You can see that if there is only one possible class a given call can make, the number of classes used in the program doesn’t matter. Another implication is that if you are comparing two alternatives via a common interface, you have to be careful how you run tests so that the first run is not favoured by the fact only one type has been used. To address this I suggest; running all the tests multiple times for at least 2 seconds in total. This should reduce the bais associated with the order you perform the tests.   Reference: Performance of inlined virtual method invocations in Java from our JCG partner Peter Lawrey at the Vanilla Java blog. ...
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below: