Featured FREE Whitepapers

What's New Here?


Turbo-charge your Android emulator for faster development

I came across an article, which claims to boost the Android emulator’s performance using Intel’s Hardware Accelerated Execution Manager (HAXM) driver. It got me excited and I decided to verify this claim. This blog entry is my story. My tools:Android SDK r20.0.3 Intellij Community Edition 11.1.3Basically, the special ‘enhancement’ provided by Intel is a special x86 Atom system image which utilizes the HAXM driver that enables better emulator performance. I’ll not repeat the technical details here, you can access the links below for more info. Caveat: This trick only works on Intel hardware and with the Virtualization Technology for Directed I/O (VT-d) enabled (usually via BIOS). Also, Intel x86 system images are currently (as of this blog posting) available for Android versions 2.3.3 (Gingerbread), 4.0.3 (ICD), and 4.1 (Jelly Bean) only. To avoid headaches, set the environment variable ANDROID_SDK_HOME to point to your Android SDK root folder before proceeding. High-level steps:  1. Download & install relevant packages via Android SDK Manager 2. Create Android Virtual Devices (AVD) 3. Create an Android Module project in IntelliJ CE 4. Test launching the Android application using the AVDs 1. Download relevant packages via Android SDK Manager Launch the SDK Manager and ensure the following is installed:Intel x86 Atom System Images (shown below is for Android 2.3.3) Intel x86 Emulator Accelerator (HAXM)Next, you’ll need to install the HAXM driver manually. Go to the Android SDK root folder and navigate to extras\intel\Hardware_Accelerated_Execution_Manager. Execute file IntelHaxm.exe to install. 2. Create Android Virtual Devices (AVD) Launch the AVD Manager and create 2 AVDs with the same options but different Target:DefaultAVD233 – Android 2.3.3 – API Level 10 IntelAVD233 – Intel Atom x86 System Image (Intel Corporation) – API Level 103. Create an Android Module project in IntelliJ CE In IntelliJ, create a new project of type ‘Android Module’, as shown:Under ‘Android SDK’, select the appropriate Android platform. You’ll need to point to your Android SDK root folder in order to choose the appropriate build target. As shown below, ‘Android 2.3.3′ is chosen:Ensure that the ‘Target Device’ option is set to Emulator, then click ‘Finish’ to complete the project creation. 4. Test launching the Android application using the AVDs Ok, we’ll test using the default Android 2.3.3 AVD first. At the IntelliJ menubar, select ‘Run’ > ‘Edit Configurations…’. Go to the ‘Target Device’ section. At the ‘Prefer Android Virtual Device’ option, select ‘DefaultAVD233′. Then Run the Android application. After a while, you should see the emulator window with the ‘Hello World’ message. To run with the Intel AVD, choose the ‘IntelAVD233′ instead. What’s most exciting is the speed of the emulator launch (timed from clicking ‘Run’ in IntelliJ up to the ‘Hello World’ message is shown in the emulator). The rough timings recorded using my notebook (Intel i3 380M, 3GB RAM):DefaultAVD233 – 1m 7s IntelAVD233 – 35sWow, that’s fast (~50% faster), without tuning other parameters to speed things up even further. Reference: Turbo-charge your Android emulator for faster development from our JCG partner Allen Julia at the YK’s Workshop blog....

Java 7: Meet the Fork/Join Framework

JSR-166(y) is the official name of this new feature which is included in Java 7. If you notice there is a ‘y’ in the name, this is because JSR-166 (Concurrency Utilities) is being added since Java 5, but it wont stop here as there are already plans to add new classes in Java 8 under the JSR-166(e). Check this page maintained by Doug Lea, the creator of JSR-166, for more information. According to Wikipedia, Parallelism is the ‘simultaneous execution of some combination of multiple instances of programmed instructions and data on multiple processors’ and Java has classes and interfaces to achieve this (sort of…) since DAY 1. You may know them as: java.lang.Thread, java.lang.Runnable, etc… What Concurrency Utilities ( java.util.concurrent package) does is simplify the way we code concurrent tasks, so our code is much simpler and cleaner. As developers we haven’t had to do anything when running our applications in machines with higher processing resources, obviously, the performance of our applications will improve, but are we really using the processing resources to the maximum? The answer is big NO. This post will show you how the Fork/Join framework will help us in using the processing resources to the maximum when dealing with problems that can be divided into small problems and all the solutions to each one of those small problems produce the solution of the big problem (like recursion, divide and conquer). What you need NetBeans 7+ or any other IDE that supports Java 7 JDK 7+ Blur on an image, example from OracleThe Basics The Fork/Join framework focuses on using all the processing resources available in the machine to improve the performance of the applications. It was designed to simplify parallelism in Divide and Conquer algorithms. The magic behind the Fork/Join framework is its work-stealing algorithm in which work threads that are free steal tasks from other busy threads, so all threads are working at all times. Following are the basics you should know in order to start using the framework:Fork means splitting the task into subtasks and work on them. Join means merging the solution of every subtask into one general solution. java.lang.Runtime use this class in order to obtain the number of processors available to the Java virtual machine. Use the method +availableProcessors():int in order to do so. java.util.concurrent.ForkJoinPool Main class of the framework, is the one that implements the work-stealing algorithm and is responsible for running the tasks. java.util.concurrent.ForkJoinTask Abstract class for the tasks that run in a java.util.concurrent.ForkJoinPool. Understand a task as a portion of the whole work, for example, if you need to to do something on an array, one task can work on positions 0 to n/2 and another task can work on positions (n/2) +1 to n-1, where n is the length of the array.java.util.concurrent.RecursiveAction Subclass of the abstract task class, use it when you don’t need the task to return a result, for example, when the task works on positions of an array, it doesn’t return anything because it worked on the array. The method you should implement in order to do the job is compute():void, notice the void return. java.util.concurrent.RecursiveTask Subclass of the abstract task class, use it when your tasks return a result. For example, when computing Fibonacci numbers, each task must return the number it computed in order to join them and obtain the general solution. The method you should implement in order to do the job is compute():V, where V is the type of return; for the Fibonacci example, V may be java.lang.Integer.When using the framework, you should define a flag that indicates whether it is necessary to fork/join the tasks or whether you should compute the work directly. For example, when working on an array, you may specify that if the length of the array is bigger than 500_000_000 you should fork/join the tasks, otherwise, the array is small enough to compute directly. In essence, the algorithm you should follow is shown next: if(the job is small enough) { compute directly } else { split the work in two pieces (fork) invoke the pieces and join the results (join) } OK, too much theory for now, let’s review an example. The Example Blurring an image requires to work on every pixel of the image. If the image is big enough we are going to have a big array of pixels to work on and so we can use fork/join to work on them and use the processing resources to the maximum. You can download the source code from the Java™ Tutorials site. Once you download the source code, open NetBeans IDE 7.x and create a new project:Then select Java Project with Existing Sources from the Java category in the displayed pop-up window:              Select a name and a project folder and click Next >Now, select the folder where you downloaded the source code for the Blur on an image example:And select the file ForkBlur.java then click finish:The source code will be imported and a new project will be created. Notice that the new project is shown with erros, this is because Java 7 is not enable for default:To fix this, right click on the project name and select the option Properties. On the pop-up dialog, go to Libraries and select JDK 1.7 from the Java Platform ComboBox:Now, go to the option Sources and select JDK 7 from the Source/Binary Format ComboBox:Last but not least, increase the memory assigned to the virtual machine when running this application as we’ll be accessing a 5 million positions array (or more). Go to the option Run and insert -Xms1024m -Xmx1024m on the VM Options TextBox:Click OK and your project should be compiling with no errors. Now, we need to find an image bigger enough so we can have a large array to work on. After a while, I found some great images (around 150 MB) from planet Mars, thanks to the curiosity robot, you can download yours from here. Once you download the image, past it on the project’s folder. Before we run the example, we need to modify the source code in order to control when to run it using the Fork/Join framework. In the ForkBlur.java file, go to line 104 in order to change the name of the image that we are going to use: //Change for the name of the image you pasted //on the project's folder. String filename = 'red-tulips.jpg'; Then, replace lines 130 to 136 with the following piece of code: ForkBlur fb = new ForkBlur(src, 0, src.length, dst); boolean computeDirectly = true;long startTime = System.currentTimeMillis(); if (computeDirectly) { fb.computeDirectly(); } else { ForkJoinPool pool = new ForkJoinPool(); pool.invoke(fb); } long endTime = System.currentTimeMillis(); Notice the computeDirectly flag. When true, we’ll NOT be using the fork/Join Framework, instead we will compute the task directly. When false, the fork/join framework will be used. The compute():void method in the ForkBlur class implements the fork/join algorithm. It’s based on the length of the array, when the length of the array is bigger than 10_000, the task will be forked, otherwise, the task will be computed directly. Following you can see my 2 processors when executing the Blur on an image example without using the Fork/Join framework ( computeDirectly = true), it took about 14s to finish the work:You can see that the processors are working, but not to the maximum. When using the Fork/Join framework ( computeDirectly = false) you can see them working at 100% and it took almost 50% less time to finish the work:This video shows the complete process: I hope you can see how useful this framework is. Of course, you cannot use it all around your code, but whenever you have a task that can be divided into small tasks then you know who to call. Reference: Java 7: Meet the Fork/Join Framework from our JCG partner Alexis Lopez at the Java and ME blog....

Distributed Apache Flume Setup With an HDFS Sink

I have recently spent a few days getting up to speed with Flume, Cloudera‘s distributed log offering. If you haven’t seen this and deal with lots of logs, you are definitely missing out on a fantastic project. I’m not going to spend time talking about it because you can read more about it in the users guide or in the Quora Flume Topicin ways that are better than I can describe it. But I will tell you about is my experience setting up Flume in a distributed environment to sync logs to a HDFS sink.Context I have 3 kinds of servers all running Ubuntu 10.04 locally: hadoop-agent-1: This is the agent which is producing all the logs hadoop-collector-1: This is the collector which is aggregating all the logs (from hadoop-agent-1, agent-2, agent-3, etc) hadoop-master-1: This is the flume master node which is sending out all the commands To add the CDH3 repository: Create a new file /etc/apt/sources.list.d/cloudera.list with the following contents: deb http://archive.cloudera.com/debian <RELEASE>-cdh3 contrib deb-src http://archive.cloudera.com/debian <RELEASE>-cdh3 contrib where: is the name of your distribution, which you can find by running lsb_release -c. For example, to install CDH3 for Ubuntu Lucid, use lucid-cdh3 in the command above. (To install a different version of CDH on a Debian system, specify the version number you want in the -cdh3 section of the deb command. For example, to install CDH3 Update 0 for Ubuntu Maverick, use maverick-cdh3u0 in the command above.) (Optionally) add a repository key. Add the Cloudera Public GPG Key to your repository by executing the following command: $ curl -s http://archive.cloudera.com/debian/archive.key | sudo apt-key add - This key enables you to verify that you are downloading genuine packages Initial Setup On both hadoop-agent-1 and hadoop-collector-1, you’ll have to install flume-node (flume-node contains the files necessary to run the agent or the collector). sudo apt-get update sudo apt-get install flume-node On hadoop-master-1: sudo apt-get update sudo apt-get install flume-master First let’s jump onto the agent and set that up. Tune the hadoop-master-1 and hadoop-collector-1 variables appropriately, but change your /etc/flume/conf/flume-site.xml to look like: <configuration> <property> <name>flume.master.servers</name> <value>hadoop-master-1</value> <description>This is the address for the config servers status server (http)</description> </property><property> <name>flume.collector.event.host</name> <value>hadoop-collector-1</value> <description>This is the host name of the default 'remote' collector.</description> </property><property> <name>flume.collector.port</name> <value>35853</value> <description>This default tcp port that the collector listens to in order to receive events it is collecting.</description> </property><property> <name>flume.agent.logdir</name> <value>/tmp/flume-${user.name}/agent</value> <description> This is the directory that write-ahead logging data or disk-failover data is collected from applications gets written to. The agent watches this directory. </description> </property> </configuration> Now on to the collector. Same file, different config. <configuration> <property> <name>flume.master.servers</name> <value>hadoop-master-1</value> <description>This is the address for the config servers status server (http)</description> </property><property> <name>flume.collector.event.host</name> <value>hadoop-collector-1</value> <description>This is the host name of the default 'remote' collector.</description> </property><property> <name>flume.collector.port</name> <value>35853</value> <description>This default tcp port that the collector listens to in order to receive events it is collecting.</description> </property><property> <name>fs.default.name</name> <value>hdfs://hadoop-master-1:8020</value> </property><property> <name>flume.agent.logdir</name> <value>/tmp/flume-${user.name}/agent</value> <description> This is the directory that write-ahead logging data or disk-failover data is collected from applications gets written to. The agent watches this directory. </description> </property><property> <name>flume.collector.dfs.dir</name> <value>file:///tmp/flume-${user.name}/collected</value> <description>This is a dfs directory that is the the final resting place for logs to be stored in. This defaults to a local dir in /tmp but can be hadoop URI path that such as hdfs://namenode/path/ </description> </property><property> <name>flume.collector.dfs.compress.gzip</name> <value>true</value> <description>Writes compressed output in gzip format to dfs. value is boolean type, i.e. true/false</description> </property><property> <name>flume.collector.roll.millis</name> <value>60000</value> <description>The time (in milliseconds) between when hdfs files are closed and a new file is opened (rolled). </description> </property> </configuration> Web Based Setup I chose to do the individual machine setup via the master web interface. You can get to this pointing your web browser at http://hadoop-master-1:35871/ (replace hadoop-master-1 with public/private DNS IP of your flume master or setup /etc/hosts for a hostname). Ensure that the port is accessible from the outside through your security settings. At this point, it was easiest for me to ensure all hosts running flume could talk to all ports on all other hosts running flume. You can certainly lock this down to the individual ports for security once everything is up and running. At this point, you should go to hadoop-agent-1 and hadoop-collector-1 run /etc/init.d/flume-node start. If everything goes well, then the master (whose IP is specified in their configs) should be notified of their existence. Now you can configure them from the web. Click on the config link and then fill in the text lines as follows (use what is in bold): Agent Node: hadoop-agent-1 Source: tailDir(“/var/logs/apache2/”,”.*.log”) Sink: agentBESink(“hadoop-collector-1?,35853) Note: I chose to use tailDir since I will control rotating the logs on my own. I am also using agentBESink because I am ok with losing log lines if the case arises. Now click Submit Query and go back to the config page to setup the collector: Agent Node: hadoop-collector-1 Source: collectorSource(35853) Sink: collectorSink(“hdfs://hadoop-master-1:8020/flume/logs/%Y/%m/%d/%H00?,”server”) This is going to tell the collector that we are sinking to HDFS with the with an initial folder of ‘flume’. It will then log to sub-folders with “flume/logs/YYYY/MM/DD/HH00? (or 2011/02/03/1300/server-.log). Now click Submit Query and go to the ‘master’ page and you should see 2 commands listed as “SUCCEEDED” in the command history. If they have not succeeded, ensure a few things have been done (there are probably more, but this is a handy start: Always use double quotes (“) since single quotes (‘) aren’t interpreted correctly. UPDATE: Single quotes are interpreted correctly, they are just not accepted intentionally (Thanks jmhsieh) In your regex, use something like “.*\\.log” since the ‘.’ is part of the regex. In your regex, ensure that your blackslashes are properly escaped: “foo\\bar” is the correct version of trying to match “foo\bar”. Additionally, there are also tables of Node Status and Node Configuration. These should match up with what you think you configured. At this point everything should work. Admittedly I had a lot of trouble getting to this point. But with the help of the Cloudera folks and the users on irc.freenode.net in #flume, I was able to get things going. The logs sadly aren’t too helpful here in most cases (but look anyway cause they might provide you with more info than they provided for me). If I missed anything in this post or there is something else I am unaware of, then let me know. Reference: Distributed Apache Flume Setup With an HDFS Sink from our JCG partner Evan Conkle at the Evan Conkle’s blog blog....

Start Android Development in 5 Minutes

Android OS is the most popular mobile OS nowadays (leading market with %46.9 above iOS’s %18.8). Besides, globally over %20 of computers are mobile devices. So, over %10 of people in the world uses Android OS and softwares related with it. For this reason, Android application development is a very important issue for developers and software engineers.This post will describe a fast (5 minutes, excluding tool download periods) and simple startup of an android project on Eclipse IDE, for developers who deal with the matter. 1. JDK and Eclipse IDE: Download and install appropriate JDK for your computer and OS (JDK 6 or 7 are preferred) http://www.oracle.com/technetwork/java/javase/downloads/index.html Download latest Eclipse distribution, unzip it to your computer. http://www.eclipse.org/downloads/2. Android SDK: Download and install android SDK: http://developer.android.com/sdk/index.html Run Android SDK and install some API (e.g. 8, 14, 16, …). Android distributions have various APIs and your application API must be compatible with your device. This step may also be performed using Eclipse ADT plug-in Android toolbar buttons. For each API, ‘SDK Platform’ is required. ‘SDK Tools’ and ‘SDK Platform Tools’ under ‘Tools’ menu and ‘Android Support Library’ under ‘Extras’ menu should be installed for general usage.Don’t forget to set your Android SDK directory on Eclipse Window –> Preferences:3. Eclipse ADT Plugin: Start Eclipse. Click Help –> Install New Software, paste adress below and click OK: https://dl-ssl.google.com/android/eclipse/Select ‘Android Development Tools’ from ‘Developer Tools’, click ‘Next’ various times and finish installation. You will see Android SDK (which performs installing different APIs as told in step 2) and Android emulator buttons on the top panel. You can also start Android projects from now on.4. Android Emulator: Android SDK includes a useful emulator for testing. After installing ADT plug-in, you can start emulator by ‘Android Virtual Device Manageer’ button from toolbar. After that you may add one or more android devices with different API and system configurations.You can have detailed info about emulator usage and command line parameters here: http://developer.android.com/tools/help/emulator.html 5. Creating Project: Select File –> New –> Project –> Android Application Project. You can also start an Android Sample Project for exercising.You must select ‘Build SDK’ and ‘Minimum Required SDK’ while creating project.After that right click your project and select ‘Run’. Your project will run on selected emulator. After this short and simple start, you may want to look at these: http://developer.android.com/training/basics/firstapp/index.html http://www.vogella.com/articles/Android/article.html http://www.coreservlets.com/android-tutorial/ http://java.dzone.com/articles/10-attractive-android http://www.javacodegeeks.com/2010/10/android-full-application-tutorial.html Don’t forget to share! Reference: Start Android Development in 5 Minutes from our JCG partner Cagdas Basaraner at the CodeBuild blog....

MapReduce: Working Through Data-Intensive Text Processing – Local Aggregation Part II

This post continues with the series on implementing algorithms found in the Data Intensive Processing with MapReduce book. Part one can be found here. In the previous post, we discussed using the technique of local aggregation as a means of reducing the amount of data shuffled and transferred across the network. Reducing the amount of data transferred is one of the top ways to improve the efficiency of a MapReduce job. A word-count MapReduce job was used to demonstrate local aggregation. Since the results only require a total count, we could re-use the same reducer for our combiner as changing the order or groupings of the addends will not affect the sum. But what if you wanted an average? Then the same approach would not work because calculating an average of averages is not equal to the average of the original set of numbers. With a little bit of insight though, we can still use local aggregation. For these examples we will be using a sample of the NCDC weather dataset used in Hadoop the Definitive Guide book. We will calculate the average temperature for each month in the year 1901. The averages algorithm for the combiner and the in-mapper combining option can be found in chapter 3.1.3 of Data-Intensive Processing with MapReduce. One Size Does Not Fit All Last time we described two approaches for reducing data in a MapReduce job, Hadoop Combiners and the in-mapper combining approach. Combiners are considered an optimization by the Hadoop framework and there are no guarantees on how many times they will be called, if at all. As a result, mappers must emit data in the form expected by the reducers so if combiners aren’t involved, the final result is not changed. To adjust for calculating averages, we need to go back to the mapper and change it’s output.Mapper Changes In the word-count example, the non-optimized mapper simply emitted the word and the count of 1. The combiner and in-mapper combining mapper optimized this output by keeping each word as a key in a hash map with the total count as the value. Each time a word was seen the count was incremented by 1. With this setup, if the combiner was not called, the reducer would receive the word as a key and a long list of 1?s to add together, resulting in the same output (of course using the in-mapper combining mapper avoided this issue because it’s guaranteed to combine results as it’s part of the mapper code). To compute an average, we will have our base mapper emit a string key (the year and month of the weather observation concatenated together) and a custom writable object, called TemperatureAveragingPair. The TemperatureAveragingPair object will contain two numbers (IntWritables), the temperature taken and a count of one. We will take the MaximumTemperatureMapper from Hadoop: The Definitive Guide and use it as inspiration for creating an AverageTemperatureMapper: public class AverageTemperatureMapper extends Mapper<LongWritable, Text, Text, TemperatureAveragingPair> { //sample line of weather data //0029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF10899199999999999private Text outText = new Text(); private TemperatureAveragingPair pair = new TemperatureAveragingPair(); private static final int MISSING = 9999;@Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); String yearMonth = line.substring(15, 21);int tempStartPosition = 87;if (line.charAt(tempStartPosition) == '+') { tempStartPosition += 1; }int temp = Integer.parseInt(line.substring(tempStartPosition, 92));if (temp != MISSING) { outText.set(yearMonth); pair.set(temp, 1); context.write(outText, pair); } } } By having the mapper output a key and TemperatureAveragingPair object our MapReduce program is guaranteed to have the correct results regardless if the combiner is called.Combiner We need to reduce the amount of data sent, so we will sum the temperatures, and sum the counts and store them separately. By doing so we will reduce data sent, but preserve the format needed for calculating correct averages. If/when the combiner is called, it will take all the TemperatureAveragingPair objects passed in and emit a single TemperatureAveragingPair object for the same key, containing the summed temperatures and counts. Here is the code for the combiner: public class AverageTemperatureCombiner extends Reducer<Text,TemperatureAveragingPair,Text,TemperatureAveragingPair> { private TemperatureAveragingPair pair = new TemperatureAveragingPair();@Override protected void reduce(Text key, Iterable<TemperatureAveragingPair> values, Context context) throws IOException, InterruptedException { int temp = 0; int count = 0; for (TemperatureAveragingPair value : values) { temp += value.getTemp().get(); count += value.getCount().get(); } pair.set(temp,count); context.write(key,pair); } } But we are really interested in being guaranteed we have reduced the amount of data sent to the reducers, so we’ll have a look at how to achieve that next.In Mapper Combining Averages Similar to the word-count example, for calculating averages, the in-mapper-combining mapper will use a hash map with the concatenated year+month as a key and a TemperatureAveragingPair as the value. Each time we get the same year+month combination, we’ll take the pair object out of the map, add the temperature and increase the count by by one. Once the cleanup method is called we’ll and emit all pairs with their respective key: public class AverageTemperatureCombiningMapper extends Mapper<LongWritable, Text, Text, TemperatureAveragingPair> { //sample line of weather data //0029029070999991901010106004+64333+023450FM-12+000599999V0202701N015919999999N0000001N9-00781+99999102001ADDGF10899199999999999private static final int MISSING = 9999; private Map<String,TemperatureAveragingPair> pairMap = new HashMap<String,TemperatureAveragingPair>();@Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); String yearMonth = line.substring(15, 21);int tempStartPosition = 87;if (line.charAt(tempStartPosition) == '+') { tempStartPosition += 1; }int temp = Integer.parseInt(line.substring(tempStartPosition, 92));if (temp != MISSING) { TemperatureAveragingPair pair = pairMap.get(yearMonth); if(pair == null){ pair = new TemperatureAveragingPair(); pairMap.put(yearMonth,pair); } int temps = pair.getTemp().get() + temp; int count = pair.getCount().get() + 1; pair.set(temps,count); } }@Override protected void cleanup(Context context) throws IOException, InterruptedException { Set<String> keys = pairMap.keySet(); Text keyText = new Text(); for (String key : keys) { keyText.set(key); context.write(keyText,pairMap.get(key)); } } } By following the same pattern of keeping track of data between map calls, we can achieve reliable data reduction by implementing an in-mapper combining strategy. The same caveats apply for keeping state across all calls to the mapper, but looking at the gains that can be made in processing efficiency, using this approach merits some consideration.Reducer At this point, writing our reducer is easy, take a list of pairs for each key, sum all the temperatures and counts then divide the sum of the temperatures by the sum of the counts. public class AverageTemperatureReducer extends Reducer<Text, TemperatureAveragingPair, Text, IntWritable> { private IntWritable average = new IntWritable();@Override protected void reduce(Text key, Iterable<TemperatureAveragingPair> values, Context context) throws IOException, InterruptedException { int temp = 0; int count = 0; for (TemperatureAveragingPair pair : values) { temp += pair.getTemp().get(); count += pair.getCount().get(); } average.set(temp / count); context.write(key, average); } }Results The results are predictable with the combiner and in-mapper-combining mapper options showing substantially reduced data output. Non-Optimized Mapper Option: 12/10/10 23:05:28 INFO mapred.JobClient: Reduce input groups=12 12/10/10 23:05:28 INFO mapred.JobClient: Combine output records=0 12/10/10 23:05:28 INFO mapred.JobClient: Map input records=6565 12/10/10 23:05:28 INFO mapred.JobClient: Reduce shuffle bytes=111594 12/10/10 23:05:28 INFO mapred.JobClient: Reduce output records=12 12/10/10 23:05:28 INFO mapred.JobClient: Spilled Records=13128 12/10/10 23:05:28 INFO mapred.JobClient: Map output bytes=98460 12/10/10 23:05:28 INFO mapred.JobClient: Total committed heap usage (bytes)=269619200 12/10/10 23:05:28 INFO mapred.JobClient: Combine input records=0 12/10/10 23:05:28 INFO mapred.JobClient: Map output records=6564 12/10/10 23:05:28 INFO mapred.JobClient: SPLIT_RAW_BYTES=108 12/10/10 23:05:28 INFO mapred.JobClient: Reduce input records=6564 Combiner Option: 12/10/10 23:07:19 INFO mapred.JobClient: Reduce input groups=12 12/10/10 23:07:19 INFO mapred.JobClient: Combine output records=12 12/10/10 23:07:19 INFO mapred.JobClient: Map input records=6565 12/10/10 23:07:19 INFO mapred.JobClient: Reduce shuffle bytes=210 12/10/10 23:07:19 INFO mapred.JobClient: Reduce output records=12 12/10/10 23:07:19 INFO mapred.JobClient: Spilled Records=24 12/10/10 23:07:19 INFO mapred.JobClient: Map output bytes=98460 12/10/10 23:07:19 INFO mapred.JobClient: Total committed heap usage (bytes)=269619200 12/10/10 23:07:19 INFO mapred.JobClient: Combine input records=6564 12/10/10 23:07:19 INFO mapred.JobClient: Map output records=6564 12/10/10 23:07:19 INFO mapred.JobClient: SPLIT_RAW_BYTES=108 12/10/10 23:07:19 INFO mapred.JobClient: Reduce input records=12 In-Mapper-Combining Option: 12/10/10 23:09:09 INFO mapred.JobClient: Reduce input groups=12 12/10/10 23:09:09 INFO mapred.JobClient: Combine output records=0 12/10/10 23:09:09 INFO mapred.JobClient: Map input records=6565 12/10/10 23:09:09 INFO mapred.JobClient: Reduce shuffle bytes=210 12/10/10 23:09:09 INFO mapred.JobClient: Reduce output records=12 12/10/10 23:09:09 INFO mapred.JobClient: Spilled Records=24 12/10/10 23:09:09 INFO mapred.JobClient: Map output bytes=180 12/10/10 23:09:09 INFO mapred.JobClient: Total committed heap usage (bytes)=269619200 12/10/10 23:09:09 INFO mapred.JobClient: Combine input records=0 12/10/10 23:09:09 INFO mapred.JobClient: Map output records=12 12/10/10 23:09:09 INFO mapred.JobClient: SPLIT_RAW_BYTES=108 12/10/10 23:09:09 INFO mapred.JobClient: Reduce input records=12 Calculated Results: (NOTE: the temperatures in the sample file are in Celsius * 10)Non-Optimized Combiner In-Mapper-Combiner Mapper190101 -25 190102 -91 190103 -49 190104 22 190105 76 190106 146 190107 192 190108 170 190109 114 190110 86 190111 -16 190112 -77 190101 -25 190102 -91 190103 -49 190104 22 190105 76 190106 146 190107 192 190108 170 190109 114 190110 86 190111 -16 190112 -77 190101 -25 190102 -91 190103 -49 190104 22 190105 76 190106 146 190107 192 190108 170 190109 114 190110 86 190111 -16 190112 -77Conclusion We have covered local aggregation for both the simple case where one could reuse the reducer as a combiner and a more complicated case where some insight on how to structure the data while still gaining the benefits of locally aggregating data for increase processing efficiency.Further ReadingData-Intensive Processing with MapReduce by Jimmy Lin and Chris Dyer Hadoop: The Definitive Guide by Tom White Source Code from blog Hadoop API MRUnit for unit testing Apache Hadoop map reduce jobs Project Gutenberg a great source of books in plain text format, great for testing Hadoop jobs locally.Reference: Working Through Data-Intensive Text Processing with MapReduce – Local Aggregation Part II from our JCG partner Bill Bejeck at the Random Thoughts On Coding blog....

CometD: Facebook similar chat for your Java web application

Chatting is easy just like eating a piece of cake or drinking a hot coffee. Have you ever thought about developing a chat program by yourself?. You know that, it is not easy as chatting. But, if you are a developer and if you read to the end of this article, you may put a try to develop a chatting application by your self and allow your users to chat via your web application.I had to implement a chatting application for my web application. As every one does, I started to search on internet. I found IRC. When I read and search more about IRC, I understood that finding a web base client for IRC was difficult. I wanted to have more customizable web client which is working similar to Facebook. At last and luckily, I found CometD. Finally, I was able to implement chatting application by using CometD and more customizable chat windows opening on the browser which is exactly similar to Facebook. This works almost all the modern browsers. This article explains step by step, How to implement chatting application from the scratch and also How to integrate chatting application to your existing Java base web application. Remember, Your web application should be a Java base one. You need to download the cometD from their official web site. It has all the dependencies required to implement the chatting application except tow java script libraries. I have written two Javascript libraries, one to create dynamic chat windows like Facebook and other to handle CometD chatting functionality in generic way. If you can manage these stuff by your self, you don’t need to use those tow Javascript libraries. Actually, CometD documentation provides good details. But, I go ahead with the tutorial by using those tow libraries. Any way, I recommend first use those tow libraries and then customize it as you need. I hope to share the sample application with you and you can deploy it in your localhost and test, how it works. 1.Adding required jar files. If you use maven to build your project, add the following dependencies into your pom.xml file <dependencies> <dependency> <groupId>org.cometd.java</groupId> <artifactId>bayeux-api</artifactId> <version>2.5.0</version> </dependency> <dependency> <groupId>org.cometd.java</groupId> <artifactId>cometd-java-server</artifactId> <version>2.5.0</version> </dependency> <dependency> <groupId>org.cometd.java</groupId> <artifactId>cometd-websocket-jetty</artifactId> <version>2.5.0</version> <exclusions> <exclusion> <groupId>org.cometd.java</groupId> <artifactId>cometd-java-client</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-simple</artifactId> <version>1.6.6</version> </dependency> <dependency> <groupId>org.cometd.java</groupId> <artifactId>cometd-java-annotations</artifactId> <version>2.5.0</version> </dependency> </dependencies> If you are not using maven to build your project, just copy the following .jar files into /WEB-INF/lib folder from your CometD download bundle. You can find these .jar files from /cometd-demo/target/cometd-demo-2.5.0.warfile.bayeux-api-2.5.0.jar cometd-java-annotations-2.5.0.jar cometd-java-common-2.5.0.jar cometd-java-server-2.5.0.jar cometd-websocket-jetty-2.5.0.jar javax.inject-1.jar jetty-continuation-7.6.7.v20120910.jar jetty-http-7.6.7.v20120910.jar jetty-io-7.6.7.v20120910.jar jetty-jmx-7.6.7.v20120910.jar jetty-util-7.6.7.v20120910.jar jetty-websocket-7.6.7.v20120910.jar jsr250-api-1.0.jar slf4j-api-1.6.6.jar slf4j-simple-1.6.6.jar2.Adding required Javascript files. You need to link the following Javascript files.cometd.js AckExtension.js ReloadExtension.js jquery-1.8.2.js jquery.cookie.js jquery.cometd.js jquery.cometd-reload.js chat.window.js comet.chat.jsThe ‘ chat.window.js‘ and ‘ comet.chat.js‘ are my own tow Javascript libraries which does not come with CometD distribution. If you are totally following this tutorial, you have to link those tow libraries as well. Provided sample application has these tow Javascript libraries. 3.Writing chat service class. /** * @author Semika siriwardana * CometD chat service. */ package com.semika.cometd;import java.util.HashMap; import java.util.Map; import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap;import javax.inject.Inject;import org.cometd.annotation.Configure; import org.cometd.annotation.Listener; import org.cometd.annotation.Service; import org.cometd.annotation.Session; import org.cometd.bayeux.client.ClientSessionChannel; import org.cometd.bayeux.server.BayeuxServer; import org.cometd.bayeux.server.ConfigurableServerChannel; import org.cometd.bayeux.server.ServerMessage; import org.cometd.bayeux.server.ServerSession; import org.cometd.server.authorizer.GrantAuthorizer; import org.cometd.server.filter.DataFilter; import org.cometd.server.filter.DataFilterMessageListener; import org.cometd.server.filter.JSONDataFilter; import org.cometd.server.filter.NoMarkupFilter;@Service('chat') public class ChatService { private final ConcurrentMap<String, Map<String, String>> _members = new ConcurrentHashMap<String, Map<String, String>>(); @Inject private BayeuxServer _bayeux; @Session private ServerSession _session;@Configure ({'/chat/**','/members/**'}) protected void configureChatStarStar(ConfigurableServerChannel channel) { DataFilterMessageListener noMarkup = new DataFilterMessageListener(new NoMarkupFilter(),new BadWordFilter()); channel.addListener(noMarkup); channel.addAuthorizer(GrantAuthorizer.GRANT_ALL); }@Configure ('/service/members') protected void configureMembers(ConfigurableServerChannel channel) { channel.addAuthorizer(GrantAuthorizer.GRANT_PUBLISH); channel.setPersistent(true); }@Listener('/service/members') public void handleMembership(ServerSession client, ServerMessage message) { Map<String, Object> data = message.getDataAsMap(); final String room = ((String)data.get('room')).substring('/chat/'.length()); Map<String, String> roomMembers = _members.get(room); if (roomMembers == null) { Map<String, String> new_room = new ConcurrentHashMap<String, String>(); roomMembers = _members.putIfAbsent(room, new_room); if (roomMembers == null) roomMembers = new_room; } final Map<String, String> members = roomMembers; String userName = (String)data.get('user'); members.put(userName, client.getId()); client.addListener(new ServerSession.RemoveListener() { public void removed(ServerSession session, boolean timeout) { members.values().remove(session.getId()); broadcastMembers(room, members.keySet()); } });broadcastMembers(room, members.keySet()); }private void broadcastMembers(String room, Set<String> members) { // Broadcast the new members list ClientSessionChannel channel = _session.getLocalSession().getChannel('/members/'+room); channel.publish(members); }@Configure ('/service/privatechat') protected void configurePrivateChat(ConfigurableServerChannel channel) { DataFilterMessageListener noMarkup = new DataFilterMessageListener(new NoMarkupFilter(),new BadWordFilter()); channel.setPersistent(true); channel.addListener(noMarkup); channel.addAuthorizer(GrantAuthorizer.GRANT_PUBLISH); }@Listener('/service/privatechat') protected void privateChat(ServerSession client, ServerMessage message) { Map<String,Object> data = message.getDataAsMap(); String room = ((String)data.get('room')).substring('/chat/'.length()); Map<String, String> membersMap = _members.get(room); if (membersMap == null) { Map<String,String>new_room=new ConcurrentHashMap<String, String>(); membersMap=_members.putIfAbsent(room,new_room); if (membersMap==null) membersMap=new_room; } String peerName = (String)data.get('peer'); String peerId = membersMap.get(peerName); if (peerId != null) { ServerSession peer = _bayeux.getSession(peerId); if (peer != null) { Map<String, Object> chat = new HashMap<String, Object>(); String text = (String)data.get('chat'); chat.put('chat', text); chat.put('user', data.get('user')); chat.put('scope', 'private'); chat.put('peer', peerName); ServerMessage.Mutable forward = _bayeux.newMessage(); forward.setChannel('/chat/' + room); forward.setId(message.getId()); forward.setData(chat);if (text.lastIndexOf('lazy') > 0) { forward.setLazy(true); } if (peer != client) { peer.deliver(_session, forward); } client.deliver(_session, forward); } } }class BadWordFilter extends JSONDataFilter { @Override protected Object filterString(String string) { if (string.indexOf('dang') >= 0) { throw new DataFilter.Abort(); } return string; } } } 4.Changing web.xml file. You should add the following filter into your web.xml file. <filter> <filter-name>continuation</filter-name> <filter-class>org.eclipse.jetty.continuation.ContinuationFilter</filter-class> </filter> <filter-mapping> <filter-name>continuation</filter-name> <url-pattern>/cometd/*</url-pattern> </filter-mapping> And also the following servlet. <servlet> <servlet-name>cometd</servlet-name> <servlet-class>org.cometd.annotation.AnnotationCometdServlet</servlet-class> <init-param> <param-name>timeout</param-name> <param-value>20000</param-value> </init-param> <init-param> <param-name>interval</param-name> <param-value>0</param-value> </init-param> <init-param> <param-name>maxInterval</param-name> <param-value>10000</param-value> </init-param> <init-param> <param-name>maxLazyTimeout</param-name> <param-value>5000</param-value> </init-param> <init-param> <param-name>long-polling.multiSessionInterval</param-name> <param-value>2000</param-value> </init-param> <init-param> <param-name>logLevel</param-name> <param-value>0</param-value> </init-param> <init-param> <param-name>transports</param-name> <param-value>org.cometd.websocket.server.WebSocketTransport</param-value> </init-param> <init-param> <param-name>services</param-name> <param-value>com.semika.cometd.ChatService</param-value> </init-param> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>cometd</servlet-name> <url-pattern>/cometd/*</url-pattern> </servlet-mapping> 5.Implementing client side functions. I think this section should be descriptive. If you allows your users to chat with other users, you need to show the list of online users in you web page, just like Facebook shows the online users inside the right side bar. For that, you can place a simple <span> or <div> tag inside your page. I have done it as follows. <div id='members'></div> All the online users will be displayed with in the above container. Once you click on a particular user name, it will open a new chat window similar to Facebook. For each pair of users, it will open a new chat window. To get this behaviour, you should use ‘ chat.window.js‘ which I mentioned before. Chatting in between particular pair of users will continue through a dedicated chat window. Just after user is logging into your web application as usual way, we should subscribe that user to chat channels. You can do it using the following way. $(document).ready(function(){ $.cometChat.onLoad({memberListContainerID:'members'}); }); Note that, I have passed the ‘id’ of online user list container as a configuration parameter. Then, user should be joined with channel as follows.You can call the bellow method with the username. function join(userName){ $.cometChat.join(userName); } Since for each chat, there is a dedicated chat window just like Facebook, we should maintain global Javascript array to store those created chat window objects. You need to place the following Javascript code inside your page. function getChatWindowByUserPair(loginUserName, peerUserName) { var chatWindow; for(var i = 0; i < chatWindowArray.length; i++) { var windowInfo = chatWindowArray[i]; if (windowInfo.loginUserName == loginUserName && windowInfo.peerUserName == peerUserName) { chatWindow = windowInfo.windowObj; } } return chatWindow; } function createWindow(loginUserName, peerUserName) { var chatWindow = getChatWindowByUserPair(loginUserName, peerUserName); if (chatWindow == null) { //Not chat window created before for this user pair. chatWindow = new ChatWindow(); //Create new chat window. chatWindow.initWindow({ loginUserName:loginUserName, peerUserName:peerUserName, windowArray:chatWindowArray}); //collect all chat windows opended so far. var chatWindowInfo = { peerUserName:peerUserName, loginUserName:loginUserName, windowObj:chatWindow }; chatWindowArray.push(chatWindowInfo); } chatWindow.show(); return chatWindow; } As I mentioned above, declare following global Javascript variable. var chatWindowArray = []; var config = { contextPath: '${pageContext.request.contextPath}' }; Since I am using a JSP page, I have to get the context path via ‘ pageContext‘ variable. If you are using a HTML page, manage it by your self to declare ‘config’ Javascript global variable. Now, you almost reached to last part of the tutorial. 5.How does the sample application works? You can download the comet.war file and deploy it in your server. Point the browser to following URL. http://localhost:8080/comet This will bring you to a page which has a text field and button called ‘Join’. Insert some user name as you wish and click on ‘Join’ button. Then you will be forwarded to a another page which has list of online users. Your name is highlighted in red color. To chat in your local machine, You can open another browser (IE and FF) and join to the chat channel. The peer user displays in blue color in the online users list. Once you click on a peer user, it will open a new chat window so that You can chat with him. This functions very similar to Facebook chatting. I have tested this chatting application in IE, FF and Crome and works fine. If you want any help of integrating this with your Java base web application, just send me a mail.Reference: Facebook similar chat for your Java web application. from our JCG partner Semika loku kaluge at the Code Box blog....

Create new message notification pop up in Java

First create JFrame to work as pop up. Add some JLabels in it to contain information and assign them at proper location to look like a notification message.A sample code is given below: String message = 'You got a new notification message. Isn't it awesome to have such a notification message.'; String header = 'This is header of notification message'; JFrame frame = new JFrame(); frame.setSize(300,125); frame.setLayout(new GridBagLayout()); GridBagConstraints constraints = new GridBagConstraints(); constraints.gridx = 0; constraints.gridy = 0; constraints.weightx = 1.0f; constraints.weighty = 1.0f; constraints.insets = new Insets(5, 5, 5, 5); constraints.fill = GridBagConstraints.BOTH; JLabel headingLabel = new JLabel(header); headingLabel .setIcon(headingIcon); // --- use image icon you want to be as heading image. headingLabel.setOpaque(false); frame.add(headingLabel, constraints); constraints.gridx++; constraints.weightx = 0f; constraints.weighty = 0f; constraints.fill = GridBagConstraints.NONE; constraints.anchor = GridBagConstraints.NORTH; JButton cloesButton = new JButton('X'); cloesButton.setMargin(new Insets(1, 4, 1, 4)); cloesButton.setFocusable(false); frame.add(cloesButton, constraints); constraints.gridx = 0; constraints.gridy++; constraints.weightx = 1.0f; constraints.weighty = 1.0f; constraints.insets = new Insets(5, 5, 5, 5); constraints.fill = GridBagConstraints.BOTH; JLabel messageLabel = new JLabel('<HtMl>'+message); frame.add(messageLabel, constraints); frame.setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE); frame.setVisible(true); Output of this will be:Here I have created a JFrame and add two labels; first is headingLabel which is header label and second is messageLabel which will contains message information; and a close button. I have used GridBagLayout but you can use any of your choice. Now to make this frame look like a popup we have to remove the title bar and border from this frame. For this add following line after frame.setSize(…); : frame.setUndecorated(true); Now the output will be:Note that now our frame cannot be closed as it do not have title bar close button. So to make our close button to work as frame closing button change its declaration as follows: JButton cloesButton = new JButton(new AbstractAction('x') { @Override public void actionPerformed(final ActionEvent e) { frame.dispose(); } }); After adding it you will get error as “Cannot refer to a non-final variable frame inside an inner class defined in a different method”. To get rid of this error you can adopt one of following solution:Make your frame variable as final. Make your frame variable a global variable in class. Make your class extends JFrame and remove frame variable at all.Now when you run your program it will look like same as in figure 2 but now you will be able to close your frame by clicking on closeButton. You will notice that your frame appears at the top of screen so change to its location bottom right corner of the screen add following lines after creating frame: Dimension scrSize = Toolkit.getDefaultToolkit().getScreenSize();// size of the screen Insets toolHeight = Toolkit.getDefaultToolkit().getScreenInsets(frame.getGraphicsConfiguration());// height of the task bar frame.setLocation(scrSize.width - frame.getWidth(), scrSize.height - toolHeight.bottom - frame.getHeight()); Now when you run it will look like this:Now to make it to disappear after predefined time add following lines at the end: new Thread(){ @Override public void run() { try { Thread.sleep(5000); // time after which pop up will be disappeared. frame.dispose(); } catch (InterruptedException e) { e.printStackTrace(); } }; }.start(); Up to this you have successfully created a notification pop up that will appear at bottom-right corner of screen and disappear after some time if close button is not clicked. So as a final touch you can design it as you want by applying look and feel or by applying different colours in frame. Also you can make it appear on top of all windows by adding: frame.setAlwaysOnTop(true);Some things to notice in above code blocks: 1. <HtMl> tag in messageLabel. It is to make word wrap in label. But make sure you text in message does not exceed some specific amount of length. You can adjust this and height of pop up as per your need. 2. “headingIcon” is not declared in code. It is the image icon you want to use instead of devil icon in screen shot as a heading title icon. A sample declaration will look like: ImageIcon headingIcon = new ImageIcon(“image_url”); 3. Currently a new window for our pop up is shown in task bar so if you want that no window is shown for pop up in task bar change JFrame to JDialog. 4. In above code default timeout before the pop up will be disappear is taken as 5 seconds you can update it as per your need by editing following line in code: Thread.sleep(5000); // time after which pop up will be disappeared. 5. To make close button look like default title bar’s close button “x” is taken in his text. You can write it close if you want to. Hope this helps you. Happy coding and don’t forget to share! Reference: Create new message notification pop up in Java. from our JCG partner Harsh Raval at the harryjoy blog....

How I select Open Source projects

Earlier this day somebody sent a image on twitter. Its title was something like, “how to choose Open Source projects” or so. It showed a flow chart. The first decision point was: is it an Apache project. If yes, so the creator suggests, don’t use the project. I was looking at this image and thought: wow, what complete and utter bullshit. Yes, there was some discussion in the past. Is Apache harmful? Or is it not? Some people seem to forget that GitHub is a tool. And the Apache Software Foundation is a community. The tool and the community, both have their benefits and drawbacks. But you cannot compare them 1:1. Have you ever tried to meet your github fellows? The likelihood to meet ASF guys and drink a beer together is really high. An Open Source project is more than its set of tools. But this post is not about what is better, github or ASF. Much has been said already about it (and too much bullshit). What really bugs me is that people seem to choose Open Source projects after the tools the projects use. Here is my personal list after which I choose projects. Please feel free to send me your suggestions by e-mail or as a comment, and don’t forget to +1 this postIs the project actively developed? Look at the projects source code browser. When happened the last commit? If there was one recently, did it touch the README only or is code changed? Projects which do not see an update for more than 6 months are either very, very stable (unlikely) or there is no interest anymore.Are there more people developing? One-Man-Shows might work sometimes, but if there are many people working on a project, it is unlikely that urgent bugs are not fixed. With only one man behind a project, you need to wait for fixes until he returns from vacation.Is there any support? A twitter account you can follow is not really support. You may send gists around, ok, but you can not expect in depth responses. Even Stackoverflow is not enough. Stackoverflow is good, but you need direct access to the developers sometimes. And hey, sometimes you have a stupid question when you start and well, on SO you can get downvoted.Is there any IP clearance? Where does the source code come from? From the author? Really? What, if the author has stolen it? The Apache Software Foundation has mechanisms which protect users here. On GitHub it is not so clear. But even there are some companies who might (or might not) care about this. For example. Twitter backs Bootstrap.io. My guess is this software has a clean IP. I try to use software only if I know where it comes from. This includes much Software from the ASF, some of Github or others, like jQuery.org.Are the people nice? When I use Open Source, the creators become my team mates. I usually check out the source code from most projects I use. I want to look in side. Sometimes I have questions. I don’t want to work in a team full of idiots or egomans. If the people are nice, likeliness increases that I use it.Methods How is the project developed and how is quality controlled? At Apache Commons for example there are many people around looking at every change of each component. Sometimes lenghty discussions happen if we should drop JDK 1.3 support or not. Should we change the interface now or not – it would break backwards compatibility. At Commons there are various tools in place to check binary compatibility, bug freeness and so on. Continous Integration helps to keep quality alive. Finally a Vote is required to push a release, and every of the voters does his checks. It is not easy to get out an release at the ASF, sometimes it requires 5 or more release candidates. I have seen projects on GitHub doing them same.Are there releases? Some projects on GitHub do not make releases. They ask you to check out the source code and update it, when they change things. Wow, this is a real blocker. I want tested software, I want a release number. Agile development can become Agile chaos easily.Are people speaking about it? If I have never heard of software and nobody speaks about it, I am careful. It does mean nobody else can help you if you run into problems.Are there docs? Docs? Complete? Readable? With examples? If yes then good. If no, go away. If developers don’t care on their docs, they have no real interest in their community. Good code documents itself. Right. But I cannot read everything when I am in time pressure.Is there community? Is the project only a “we stop by and drop a fix” group or is it a real community? Communities have the benefits of group effects. If somethings smells nasty, they might fork. If something is wrong in a “stop by” group the group will die.Is there one head? If the project has 1000 forks with many changes which are not found in the trunk – which one should I choose? Better is one version control system. I don’t care if it is Mercural and Bitbucket, Git and Github or SVN at the Apache Software Foundation. I am just a user in most cases, f**k, the devs spent so much time, I leave it up to them how they want to develop! It is just important that I have -one- place were I get the real, cool, official sources.Vendors Who is running the project? Is it one vendor? Are there many vendors? Or is it a collective of individuals? I am very careful when projects are backed by only one vendor. In case of Bootstrap (Twitter) I don’t care. The project is great, but so small I can replace if something wents wrong in a couple of days. If it comes to JBoss (Red Hat), I am a bit more careful. The strategy of JBoss already earned some critics back then. In this case I would prefer Apache Geronimo. This is another JEE container. At the ASF people are committing, not companies. Even when there are companies who pay committers to work fulltime on the projects (like with OpenOffice.org), there are always many other people who can continue, just in case. The ASF like having a good diversity of committers at their projects.Licenses Is the license clear and understandable? Is there a LICENSE file in the source code? I prefer the AL2.0 license (or similar, like MIT) because I can do whatever I want with it. I do lots of Open Source work, but honestly, sometimes I need to sell my software and for that AL et al works best. Projects which do not have named LICENSE, or have a complicated “double license” model or a license like the GPL usually are not usable for me.Acknowledgements Thanks to Simone Tripodi and Maurizio Cucchiara for their valuable feedback on this post.Don’t forget to share! Reference: How I select Open Source projects from our JCG partner Christian Grobmeier at the PHP und Java Entwickler blog....

On Measuring Code Coverage

In a previous post, I explained how to visualize what part of your code is covered by your tests. This post explores two questions that are perhaps more important: why and what code coverage to measure.Why We Measure Code Coverage What does it mean for a statement to be covered by tests? Well, it means that the statement was executed while the tests ran, nothing more, nothing less. We can’t automatically assume that the statement is tested, since the bare fact that a statement was executed doesn’t imply that the effects of that execution were verified by the tests. If you practice Test-First Programming, then the tests are written before the code. A new statement is added to the code only to make a failing test pass. So with Test-First Programming, you know that each executed statement is also a tested statement.If you don’t write your tests first, then all bets are off. Since Test-First Programming isn’t as popular as I think it should be, let’s assume for the remainder of this post that you’re not practicing it. Then what good does it do us to know that a statement is executed? Well, if the next statement is also executed, then we know that the first statement didn’t throw an exception. That doesn’t help us much either, however. Most statements should not throw an exception, but some statements clearly should. So in general, we still don’t get a lot of value out of knowing that a statement is executed. The true value of measuring code coverage is therefore not in the statements that are covered, but in the statements that are not covered! Any statement that is not executed while running the tests is surely not tested. Uncovered code indicates that we’re missing tests.What Code Coverage We Should Measure Our next job is to figure out what tests are missing, so we can add them. How can we do that? Since we’re measuring code coverage, we know the target of the missing tests, namely the statements that were not executed.If some of those statements are in a single class, and you have unit tests for that class, it’s easy to see that those unit tests are incomplete. Unit tests can definitely benefit from measuring code coverage. What about acceptance tests? Some code can easily be related to a single feature, so in those cases we could add an acceptance test. In general, however, the relationship between a single line of code and a feature is weak. Just think of all the code we re-use between features. So we shouldn’t expect to always be able to tell by looking at the code what acceptance test we’re missing. It makes sense to measure code coverage for unit tests, but not so much for acceptance tests.Code Coverage on Acceptance Tests Can Reveal Dead Code One thing we can do by measuring code coverage on acceptance tests, is find dead code. Dead code is code that is not executed, except perhaps by unit tests. It lives on in the code base like a zombie. Dead code takes up space, but that’s not usually a big problem. Some dead code can be detected by other means, like by your IDE. So all in all, it seems that we’re not gaining much by measuring code coverage for acceptance tests.Code Coverage on Acceptance Tests May Be Dangerous OK, so we don’t gain much by measuring coverage on acceptance tests. But no harm, no foul, right? Well, that remains to be seen. Some organizations impose targets for code coverage. Mindlessly following a rule is not a good idea, but, alas, such is often the way of big organizations. Anyway, an imposed number of, say, 75% line coverage may be achievable by executing only the acceptance tests. So developers may have an incentive to focus their tests exclusively on acceptance tests.This is not as it should be according to the Test Pyramid. Acceptance tests are slower, and, especially when working through a GUI, may also be more brittle than unit tests. Therefore, they usually don’t go much further than testing the happy path. While it’s great to know that all the units integrate well, the happy path is not where most bugs hide. Some edge and error cases are very hard to write as automated acceptance tests. For instance, how do you test what happens when the network connection drops out? These types of failures are much easier explored by unit tests, since you can use mock objects there. The path of least resistance in your development process should lead developers to do the right thing. The right thing is to have most of the tests in the form of unit tests. If you enforce a certain amount of code coverage, be sure to measure that coverage on unit tests only. Reference: On Measuring Code Coverage from our JCG partner Remon Sinnema at the Secure Software Development blog....

Master Detail CRUD operations with Regions ADF 11g

This is an example that demonstrates how to create a Master Detail relationship between tables by using Regions. The main purpose of regions is the notion of reusability. With regions and bounded task flows we can reuse our pages into many other pages keeping the same functionality and having a more cleaner approach Download the Sample Application. For this example we are going to use only one Model project and keep things simple. We are going to create our Business Components through JDeveloper and it’s wizards. We are using Master Detail for Departments and Employees.So, we are going to create two Bounded Task Flows that use fragments. One for the Departments One for the employees.In each bounded task flow we drag and drop a view and place the appropriate names of departments and employees.Then in the unbounded flow we create a jspx that will have two Regions defined. One for the Department BTF One for the Employees BTF For Departments we are going to drag and drop the Departments iterator as a form with navigation buttons and submit button. Additionally, we add the createInsert and Delete Operation buttons next to submitWe do the same with employees. The only difference here is that we drop an editable table and not a form. Additionally we drag it from the hierarchy and not the alone one in our Data Control. This means that we drag the detailed employees.Next, we are going to create an index page in our unbounded task flow that will contain our Bounded Task Flows as regions. In order to that, after we created the index page, we simply drag and drop each Bounded Task Flow as a RegionWe do the same for the Employees Bounded Task Flow. Up to now, we have our hierarchy done and well placed. Since we share the same application module instance, we are good to go!! All that is left now is to place commit and rollback buttons in our Departments fragment and we are done! For the rollback button we have to make a specific adjustment: The emps region needs to be refreshed and indicate that the rollback is performed. For this reason we are going to set the refresh property as follows:So, what we do here is, to set a refresh condition on our detail region. What we say here is, refresh emps fragment when the dept fragments is refreshed. NOTE: this is a simple application demonstrating the ease of use of Regions. It is not intended to cover all aspects of regions. Download the Sample Application. Reference: Master Detail CRUD operations with Regions ADF 11g from our JCG partner Dimitrios Stassinopoulos at the Born To DeBug blog....
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below: