Featured FREE Whitepapers

What's New Here?

mongodb-logo

MongoDB From the Trenches: Prudent Production Planning

While starting out with MongoDB is super easy, there are few things you should keep in mind as you move from a development environment into a production one. No one wants to get paged at 3am because a customer can’t complete an order on your awesome e-commerce site because your database isn’t responding fast enough or worse, is down. Planning for a production deployment with MongoDB isn’t rocket science, but I must warn you, it’ll cost money, especially if your application actually gets used a lot, which is every developer’s dream. Therefore, like all databases, you need to plan for high availability and you’ll want the maximum performance benefits you can get for your money in a production environment. First and foremost, Mongo likes memory; that is, frequently accessed data is stored directly in memory; moreover, writes are also stored in memory until being flushed to disk. It’s imperative that you provide enough memory for Mongo to store a valid working dataset; otherwise, Mongo will have to go to the disk to retrieve, what should be, fast lookups via indexed data. This is sloooooow. Therefore, a good rule of thumb is to plan to run your Mongo instances with as much memory as you can afford. You can get an idea for your working data set by running Mongostat – this is a handy command line utility that’ll give you a second-by-second view into what Mongo is up to – one particular metric you’ll see is resident memory (labeled as res) – this will give you a good idea of how much memory Mongo’s using at any given moment. If this number exceeds what you have available on a given machine, then Mongo is having to go to disk, which is going to be a lot slower. Not all data can be stored in memory; every document in Mongo is eventually written to disk. And like always, I/O is always a slow operation compared to working with memory. This is why, for example, writes in Mongo can be so fast – drivers allow you to, essentially, fire and forget and the actual write to disk is done later, asynchronously. Reads can also incur an I/O penalty when something requested isn’t in working memory. Thus, for high performance reads and writes, pay attention to the underlying disks. A key metric here is IOPS or input/output operations per second. Mongo will be extremely happy, for example, in an SSD environment, provided you can afford it. Just take a look at various IOPS comparisons between SSDs and traditional spinning disks – super fast RPM disks can achieve IOPS in the 200 range. Typical SSD drives are attaining wild numbers, orders of magnitude higher (like in the 100’s of thousands of IOPS). It’s crazy how fast SSDs are compared to traditional hard drives. RAM is still faster than SSDs, so you’ll still want to understand your working set of data and ensure you have plenty of memory to contain it. Finally, for maximum availably, you really should be using Mongo’s replica sets. Setting up a cluster of Mongo instances is so incredibility easy that there really isn’t a good reason not to do it. The benefits of doing so are manifold, including:data redundancy high availability via automated failover disaster recoveryPlus, running a replica set makes maintenance so much easier as you can bring nodes off line and on line w/out an interruption of service. And you can run nodes in a replica set on commodity hardware (don’t forget about my points regarding memory and I/O though). Accordingly, when looking to move Mongo into a production environment, you need to consider memory, I/O performance, and replica sets. Running a high performant, high availability replica set’ed Mongo, not surprisingly, will cost you. If you’re looking for options for running Mongo in a production environment, I can’t recommend enough the team at MongoHQ. I’m a huge fan of Mongo. Check out some of the articles, videos, and podcasts that I’ve done, which focus on Mongo, including:Java development 2.0: MongoDB: A NoSQL datastore with (all the right) RDBMS moves Video demo: An introduction to MongoDB Eliot Horowitz on MongoDB 10gen’s Steve Francia talks MongoDB  Reference: MongoDB From the Trenches: Prudent Production Planning from our JCG partner Andrew Glover at the The Disco Blog blog. ...
java-logo

Investigating Deadlocks – Part 3

In my previous two blogs in this series, part 1 and part 2, I’ve demonstrated how to create a piece of bad code that deadlocks and then used this code to show three ways of taking a thread dump. In this blog I’m going to analyze the thread dump to figure out what when wrong. The discussion below refers to both the Account and DeadlockDemo classes from part 1 of this series, which contains full code listings. The first thing that that I need is a thread dump from the DeadlockDemo application, so as they used to say on Blue Peter “here’s one I prepared earlier”.       2012-10-16 13:37:03 Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.10-b01-428 mixed mode):"DestroyJavaVM" prio=5 tid=7f9712001000 nid=0x110247000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE"Thread-21" prio=5 tid=7f9712944000 nid=0x118d76000 waiting for monitor entry [118d75000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366f58> (a threads.deadlock.Account) - locked <7f3366ee0> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-20" prio=5 tid=7f971216c000 nid=0x118c73000 waiting for monitor entry [118c72000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366e98> (a threads.deadlock.Account) - locked <7f3366f58> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-19" prio=5 tid=7f9712943800 nid=0x118b70000 waiting for monitor entry [118b6f000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:81) - waiting to lock <7f3366f40> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-18" prio=5 tid=7f9712942800 nid=0x118a6d000 waiting for monitor entry [118a6c000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:81) - waiting to lock <7f3366f40> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-17" prio=5 tid=7f9712942000 nid=0x11896a000 waiting for monitor entry [118969000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:81) - waiting to lock <7f3366ec8> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-16" prio=5 tid=7f9712941000 nid=0x118867000 waiting for monitor entry [118866000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:81) - waiting to lock <7f3366ec8> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-15" prio=5 tid=7f9712940800 nid=0x118764000 waiting for monitor entry [118763000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:81) - waiting to lock <7f3366ef8> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-14" prio=5 tid=7f971293f800 nid=0x118661000 waiting for monitor entry [118660000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:81) - waiting to lock <7f3366f28> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-13" prio=5 tid=7f97129ae000 nid=0x11855e000 waiting for monitor entry [11855d000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:81) - waiting to lock <7f3366eb0> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-12" prio=5 tid=7f97129ad000 nid=0x11845b000 waiting for monitor entry [11845a000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:81) - waiting to lock <7f3366f40> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-11" prio=5 tid=7f97129ac800 nid=0x118358000 waiting for monitor entry [118357000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366f58> (a threads.deadlock.Account) - locked <7f3366eb0> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-10" prio=5 tid=7f97129ab800 nid=0x118255000 waiting for monitor entry [118254000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:81) - waiting to lock <7f3366eb0> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-9" prio=5 tid=7f97129ab000 nid=0x118152000 waiting for monitor entry [118151000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366e98> (a threads.deadlock.Account) - locked <7f3366ec8> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-8" prio=5 tid=7f97129aa000 nid=0x11804f000 waiting for monitor entry [11804e000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366eb0> (a threads.deadlock.Account) - locked <7f3366f28> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-7" prio=5 tid=7f97129a9800 nid=0x117f4c000 waiting for monitor entry [117f4b000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366eb0> (a threads.deadlock.Account) - locked <7f3366e80> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-6" prio=5 tid=7f97129a8800 nid=0x117e49000 waiting for monitor entry [117e48000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:81) - waiting to lock <7f3366e80> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-5" prio=5 tid=7f97128a1800 nid=0x117d46000 waiting for monitor entry [117d45000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:81) - waiting to lock <7f3366f28> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-4" prio=5 tid=7f97121af800 nid=0x117c43000 waiting for monitor entry [117c42000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366e80> (a threads.deadlock.Account) - locked <7f3366e98> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-3" prio=5 tid=7f97121ae800 nid=0x117b40000 waiting for monitor entry [117b3f000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366e80> (a threads.deadlock.Account) - locked <7f3366ef8> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"Thread-2" prio=5 tid=7f971224a000 nid=0x117a3d000 waiting for monitor entry [117a3c000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366eb0> (a threads.deadlock.Account) - locked <7f3366f40> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)"RMI TCP Accept-0" daemon prio=5 tid=7f97128fd800 nid=0x117837000 runnable [117836000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <7f32ee740> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) at java.lang.Thread.run(Thread.java:680)"Poller SunPKCS11-Darwin" daemon prio=1 tid=7f97128fd000 nid=0x117734000 waiting on condition [117733000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:692) at java.lang.Thread.run(Thread.java:680)"Low Memory Detector" daemon prio=5 tid=7f971209e000 nid=0x1173ec000 runnable [00000000] java.lang.Thread.State: RUNNABLE"C2 CompilerThread1" daemon prio=9 tid=7f971209d000 nid=0x1172e9000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE"C2 CompilerThread0" daemon prio=9 tid=7f971209c800 nid=0x1171e6000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE"Signal Dispatcher" daemon prio=9 tid=7f971209b800 nid=0x1170e3000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE"Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7f971209a800 nid=0x116fe0000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE"Finalizer" daemon prio=8 tid=7f971209a000 nid=0x116d1c000 in Object.wait() [116d1b000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f3001300> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <7f3001300> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)"Reference Handler" daemon prio=10 tid=7f9712099000 nid=0x116c19000 in Object.wait() [116c18000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <7f30011d8> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <7f30011d8> (a java.lang.ref.Reference$Lock)"VM Thread" prio=9 tid=7f9712096800 nid=0x116b16000 runnable"Gang worker#0 (Parallel GC Threads)" prio=9 tid=7f9712002800 nid=0x1135c7000 runnable"Gang worker#1 (Parallel GC Threads)" prio=9 tid=7f9712003000 nid=0x1136ca000 runnable"Concurrent Mark-Sweep GC Thread" prio=9 tid=7f971204d800 nid=0x116790000 runnable "VM Periodic Task Thread" prio=10 tid=7f97122d4000 nid=0x11793a000 waiting on condition"Exception Catcher Thread" prio=10 tid=7f9712001800 nid=0x1103ef000 runnable JNI global references: 1037Found one Java-level deadlock: ============================= "Thread-21": waiting to lock monitor 7f97118bd560 (object 7f3366f58, a threads.deadlock.Account), which is held by "Thread-20" "Thread-20": waiting to lock monitor 7f97118bc108 (object 7f3366e98, a threads.deadlock.Account), which is held by "Thread-4" "Thread-4": waiting to lock monitor 7f9711834360 (object 7f3366e80, a threads.deadlock.Account), which is held by "Thread-7" "Thread-7": waiting to lock monitor 7f97118b9708 (object 7f3366eb0, a threads.deadlock.Account), which is held by "Thread-11" "Thread-11": waiting to lock monitor 7f97118bd560 (object 7f3366f58, a threads.deadlock.Account), which is held by "Thread-20"Java stack information for the threads listed above: =================================================== "Thread-21": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366f58> (a threads.deadlock.Account) - locked <7f3366ee0> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59) "Thread-20": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366e98> (a threads.deadlock.Account) - locked <7f3366f58> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59) "Thread-4": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366e80> (a threads.deadlock.Account) - locked <7f3366e98> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59) "Thread-7": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366eb0> (a threads.deadlock.Account) - locked <7f3366e80> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59) "Thread-11": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366f58> (a threads.deadlock.Account) - locked <7f3366eb0> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)Found 1 deadlock.Heap par new generation total 19136K, used 11590K [7f3000000, 7f44c0000, 7f44c0000) eden space 17024K, 68% used [7f3000000, 7f3b51ac0, 7f40a0000) from space 2112K, 0% used [7f40a0000, 7f40a0000, 7f42b0000) to space 2112K, 0% used [7f42b0000, 7f42b0000, 7f44c0000) concurrent mark-sweep generation total 63872K, used 0K [7f44c0000, 7f8320000, 7fae00000) concurrent-mark-sweep perm gen total 21248K, used 8268K [7fae00000, 7fc2c0000, 800000000) Scanning quickly through, you can see that this thread dump is divided into four parts. These are:A complete list of all the applcation’s threads A list of deadlocked threads A small stack trace of deadlocked threads The application’s heap summary  The Thread List The thread list in point one above is a list of all the application’s threads and their current status. From this you can see that an application consists of a whole bunch of threads, which you can roughly divide in to two. Firstly there are the background threads. These are the ones that every application has, which get on with all the dirty jobs that we, as application programmers, don’t usually need to worry about. These have names such as: “DestroyJavaVM“, Low Memory Detector, Finalizer, Exception Catcher Thread and Concurrent Mark-Sweep GC Thread. Secondly, there are the threads that you or I may create as part of our code. These usually have names that consist of the word Thread followed by a number. For example: Thread-3, Thread-6 and Thread-20. "Thread-20" prio=5 tid=7f971216c000 nid=0x118c73000 waiting for monitor entry [118c72000] java.lang.Thread.State: BLOCKED (on object monitor) at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:82) - waiting to lock <7f3366e98> (a threads.deadlock.Account) - locked <7f3366f58> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:58) Looking at the information given on Thread-20 in more detail you can see that this can be broken down into several parts. These are: <td >Thread-20 <td >The thread’s name as described above.<tr > <td >prio=5 <td >The thread’s priority. A number from 1 to 10, where 1 is the lowest and 10 is the highest priority. <tr > <tr > <td >tid=7f971216c000 <td >The thread id. A unique number that’s returned by a Thread.getId() call. <tr > <td >nid=0x118c73000 <td >The native thread id. This maps to a platform dependent thread id. <tr > <td >waiting for monitor entry [118c72000] java.lang.Thread.State: BLOCKED (on object monitor) <td >This is the status of the thread; in this case it’s BLOCKED. Also included is a stack trace outlining where the thread is blocked. Note that a thread can also be marked as a daemon. For example: “RMI TCP Accept-0″ daemon prio=5 tid=7f97128fd800 nid=0x117837000 runnable [117836000] java.lang.Thread.State: RUNNABLE Daemon threads are background task threads such as the RMI TCP Accept-0 thread listed above. A daemon thread is a thread that does not prevent the JVM from exiting. The JVM will exit, or close down, when only daemon threads remain.However, the thread list doesn’t really help in tracing the cause of a deadlock, so moving swiftly along… The Deadlock Thread List This section of the thread dump contains a list of all threads that are involved in the deadlock. Found one Java-level deadlock: ============================= "Thread-21": waiting to lock monitor 7f97118bd560 (object 7f3366f58, a threads.deadlock.Account), which is held by "Thread-20" "Thread-20": waiting to lock monitor 7f97118bc108 (object 7f3366e98, a threads.deadlock.Account), which is held by "Thread-4" "Thread-4": waiting to lock monitor 7f9711834360 (object 7f3366e80, a threads.deadlock.Account), which is held by "Thread-7" "Thread-7": waiting to lock monitor 7f97118b9708 (object 7f3366eb0, a threads.deadlock.Account), which is held by "Thread-11" "Thread-11": waiting to lock monitor 7f97118bd560 (object 7f3366f58, a threads.deadlock.Account), which is held by "Thread-20" From the segment above, you can see that there are five threads all blocking on instances the threads.deadlock.Account class Leaving aside the monitor ids and Account instances, you can see that “Thread-21″ is waiting for “Thread-20″, which is waiting for “Thread-4″, which in turn is waiting for “Thread-7″. “Thread-7″ is waiting for “Thread-11″, which is waiting for “Thread-20″: a deadlock loop as shown in the diagram below:The Deadlock Stack TracesThe final piece of the puzzle is the list of deadlocked thread stack traces as shown below: Java stack information for the threads listed above: =================================================== "Thread-21": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366f58> (a threads.deadlock.Account) - locked <7f3366ee0> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59) "Thread-20": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366e98> (a threads.deadlock.Account) - locked <7f3366f58> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59) "Thread-4": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366e80> (a threads.deadlock.Account) - locked <7f3366e98> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59) "Thread-7": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366eb0> (a threads.deadlock.Account) - locked <7f3366e80> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59) "Thread-11": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f3366f58> (a threads.deadlock.Account) - locked <7f3366eb0> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59) From the previous section, we know that Thread-20 is waiting, via a circuitous route, for Thread-11 and Thread-11 is waiting for Thread-20. This is our deadlock. The next step is to tie this deadlock up to lines of code using the thread stack trace above and I’ve simplified this in the diagram below.In the above diagram I’ve removed the 7f3366 prefix from the object ids for clarity; hence, object 7f3366f58 is now f58. From this diagram, you can see that object f58 is locked by Thread-20 on line 59 and is waiting for a lock on object e98 on line 86. Following the arrows down, you can see that Thread-7 is waiting for a lock on eb0 on line 86, which in turn is locked by Thread-11 on line 59. Thread-11 is waiting for a lock on f58 on line 86, which, looping back up, is locked on line 58 by Thread-20. So, where are these lines of code? The following shows line 59:…and this is line 86:Everyone gets surprises sometimes and the stack trace above surprised me. I was expecting the locks to be on lines 85 and 86; however, they were on 59 and 86. Since line 59 doesn’t contain a synchronized keyword, I’m guessing that the compiler has done some optimisation on the transfer(…) method’s first synchronized keyword. The conclusion that can be drawn from this is that the code, which randomly picks two Account objects from a list, is locking them in the wrong order on lines 59 and 86. So what’s the fix? More on that next time; however, there’s one final point to note, which is that the make up of a deadlock may not be the same every time you generate a thread dump on a program. After running the DeadlockDemo program again and using kill -3 PID to get hold of another thread dump, I obtained these results: Found one Java-level deadlock: ============================= "Thread-20": waiting to lock monitor 7fdc7c802508 (object 7f311a530, a threads.deadlock.Account), which is held by "Thread-3" "Thread-3": waiting to lock monitor 7fdc7a83d008 (object 7f311a518, a threads.deadlock.Account), which is held by "Thread-11" "Thread-11": waiting to lock monitor 7fdc7c802508 (object 7f311a530, a threads.deadlock.Account), which is held by "Thread-3"Java stack information for the threads listed above: =================================================== "Thread-20": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:86) - waiting to lock <7f311a530> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59) "Thread-3": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:87) - waiting to lock <7f311a518> (a threads.deadlock.Account) - locked <7f311a530> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59) "Thread-11": at threads.deadlock.DeadlockDemo$BadTransferOperation.transfer(DeadlockDemo.java:87) - waiting to lock <7f311a530> (a threads.deadlock.Account) - locked <7f311a518> (a threads.deadlock.Account) at threads.deadlock.DeadlockDemo$BadTransferOperation.run(DeadlockDemo.java:59)Found 1 deadlock. In this thread dump a smaller number of threads are involved in the deadlock, but if you analyze it you can draw the same conclusions as my first example. Next time: fixing the code… For more information see the other blogs in this series. All source code for this an other blogs in the series are available on Github at git://github.com/roghughe/captaindebug.git   Reference: Investigating Deadlocks – Part 3: Analysing the Thread Dump from our JCG partner Roger Hughes at the Captain Debug’s Blog blog. ...
junit-logo

JUnit Rules

The first time I stumbled over a JUnit @Rule annotation I was a bit irritated of the concept. Having a public field in a test case seemed somewhat odd and so I was reluctant to use it regularly. But after a while I got used to that and it turned out that rules can ease writing tests in many ways. This post gives a quick introduction of the concept and some short examples of what rules are good for. What are JUnit Rules? Let’s start with a look at a JUnit out-of-the-box rule. The TemporaryFolder is a test helper that can be used to create files and folders located under the file system directory for temporary content1. The interesting thing with the TemporaryFolder is that it guarantees to delete its files and folders when the test method finishes2. To work as expected the temporary folder instance must be assigned to an @Rule annotated field that must be public, not static, and a subtype of TestRule: public class MyTest {@Rule public TemporaryFolder temporaryFolder = new TemporaryFolder();@Test public void testRun() throws IOException { assertTrue( temporaryFolder.newFolder().exists() ); } }   How does it work? Rules provide a possibility to intercept test method calls similar as an AOP framework would do. Comparable to an around advice in AspectJ you can do useful things before and/or after the actual test execution3. Although this sounds complicated it is quite easy to achieve. The API part of a rule definition has to implement TestRule. The only method of this interface called apply returns a Statement. Statements represent – simply spoken – your tests within the JUnit runtime and Statement#evaluate() executes them. Now the basic idea is to provide wrapper extensions of Statement that can do the actual contributions by overriding Statement#evaluate(): public class MyRule implements TestRule {@Override public Statement apply( Statement base, Description description ) { return new MyStatement( base ); } }public class MyStatement extends Statement {private final Statement base;public MyStatement( Statement base ) { this.base = base; }@Override public void evaluate() throws Throwable { System.out.println( 'before' ); try { base.evaluate(); } finally { System.out.println( 'after' ); } } } MyStatement is implemented as wrapper that is used in MyRule#apply(Statement,Destination) to wrap the original statement given as argument. It is easy to see that the wrapper overrides Statement#evaluate() to do something before and after the actual evaluation of the test4. The next snippet shows how MyRule can be used exactly the same way as the TemporaryFolder above: public class MyTest {@Rule public MyRule myRule = new MyRule();@Test public void testRun() { System.out.println( 'during' ); } } Launching the test case leads to the following console output which proves that our example rule works as expected. The test execution gets intercepted and modified by our rule to print ‘before’ and ‘after’ around the ‘during’ of the test: before during after Now that the basics are understood let’s have a look at slightly more useful things you could do with rules. Test Fixtures Quoted from the according wikipedia section a test fixture ‘is all the things that must be in place in order to run a test and expect a particular outcome. Frequently fixtures are created by handling setUp() and tearDown() events of the unit testing framework’. With JUnit this often looks somewhat like this: public class MyTest {private MyFixture myFixture;@Test public void testRun1() { myFixture.configure1(); // do some testing here }@Test public void testRun2() { myFixture.configure2(); // do some testing here }@Before public void setUp() { myFixture = new MyFixture(); }@After public void tearDown() { myFixture.dispose(); } } Consider you use a particular fixture the way shown above in many of your tests. In that case it could be nice to get rid of the setUp() and tearDown() methods. Given the sections above we now know that this can be done by changing MyFixture to implement TestRule. An appropriate Statement implementation would have to ensure that it calls MyFixture#dispose() and could look like this: public class MyFixtureStatement extends Statement {private final Statement base; private final MyFixture fixture;public MyFixtureStatement( Statement base, MyFixture fixture ) { this.base = base; this.fixture = fixture; }@Override public void evaluate() throws Throwable { try { base.evaluate(); } finally { fixture.dispose(); } } } With this in place the test above can be rewritten as: public class MyTest {@Rule public MyFixture myFixture = new MyFixture();@Test public void testRun1() { myFixture.configure1(); // do some testing here }@Test public void testRun2() { myFixture.configure2(); // do some testing here } } I come to appreciate the more compact form of writing tests using rules in a lot of cases, but surely this is also a question of taste and what you consider better to read5. Fixture Configuration with Method Annotations So far I have silently ignored the Description argument of TestRule#apply(Statement,Description). In general a Description describes a test which is about to run or has been run. But it also allows access to some reflective information about the underlying java method. Among others it is possible to read the annotations attached to such a method. This enables us to combine rules with method annotations for convenience configuration of a TestRule. Consider this annotation type: @Retention(RetentionPolicy.RUNTIME) @Target({ElementType.METHOD}) public @interface Configuration { String value(); } Combined with the following snippet inside MyFixture#apply(Statement,Destination) that reads the configuration value annotated to a certain test method… Configuration annotation = description.getAnnotation( Configuration.class ); String value = annotation.value(); // do something useful with value … the test case above that demonstrates the usage of the MyFixture rule can be rewritten to: public class MyTest {@Rule public MyFixture myFixture = new MyFixture();@Test @Configuration( value = 'configuration1' ) public void testRun1() { // do some testing here }@Test @Configuration( value = 'configuration2' ) public void testRun2() { // do some testing here } } Of course there are limitations to the latter approach due to the fact that annotations only allow Enums, Classes or String literals as parameters. But there are use cases where this is completely sufficient. A nice example using rules combined with method annotations is provided by the restfuse library. If you are interested in a real world example you should have a look at the library’s implementation of the Destination rule6. Comming to the end the only thing left to say is that I would love to hear from you about other useful examples of JUnit rules you might use to ease your daily testing work:The directory which is in general returned by System.getProperty( 'java.io.tmpdir' ); ↩ Looking at the implementation of TemporaryFolder I must note that it does not check if the deletion of a file is successful. This might be a weak point in case of open file handles ↩ And for what it’s worth you even could replace the complete test method by something else ↩ The delegation to the wrapped statement is put into a try...finally block to ensure that the functionality after the test gets executed, even if the test would fail. In that case an AssertionError would be thrown and all statements that are not in the finally block would be skipped ↩ You probably noted that the TemporaryFolder example at the beginning is also nothing else but a fixture use case ↩ Note that restfuse’s Destination class implements MethodRule instead of TestRule. This post is based on the latest JUnit version where MethodRule has been marked as @Deprecated. TestRule is the replacement for MethodRule. But given the knowledge of this post it should be nevertheless easily possible to understand the implementation ↩  Reference: JUnit Rules from our JCG partner Frank Appel at the Code Affine blog. ...
android-logo

Android Reverse Engineering and Decompilation

Reverse engineering of android java app using apktool, dex2jar, jd-gui to convert .apk file to .java. By reverse engineering of android app (.apk file) we can get following :understand how a particular UI in an App is constructed reading AndroidManifest.xml – permissions, activities, intents etc in the App native libraries and images used in that App obsfucated code ( android SDK, by default, uses ProGuard tool which shrinks, optimizes, and obfuscates your code by removing unused code and renaming classes, fields, and methods with semantically obscure names.  Required Tools : Download the followings first.Dex2jar from http://code.google.com/p/dex2jar/ JD-GUI from http://java.decompiler.free.fr/?q=jdgui ApkTool from http://code.google.com/p/android-apktool/Using ApkTool - to extract AndroidManifest.xml and everything in res folder(layout xml files, images, htmls used on webview etc..) Run the following command : >apktool.bat d sampleApp.apk It also extracts the .smali file of all .class files, but which is difficult to read. ##You can achieve this by using zip utility like 7-zip. Using dex2jar - to generate .jar file from .apk file, we need JD-GUI to view the source code from this .jar. Run the following command : >dex2jar sampleApp.apk Decompiling .jar JD-GUI - it decompiles the .class files (obsfucated- in case of android app, but readable original code is obtained in case of other .jar file). i.e., we get .java back from the application. Just Run the jd-gui.exe and File->Open to view java code from .jar or .class file. You May Also Like -Android: Application Project Structure in Eclipse Final Year Computer Project Suggestion, A HUGE LIST Installing Android SDK, ADT in eclipse Java: using recursion to read a folder and its content in tree format sub-fo … Android First Program eclipse- getting started  Reference: Android Reverse Engineering – decompile .apk-.dex-.jar-.java from our JCG partner Ganesh Tiwari at the GT’s Blog blog. ...
spring-interview-questions-answers

Spring: Setting Logging Dependencies

This post describes how to set-up logging dependencies in Spring. It is based on information available in a post by Dave Syer’s. A reminder on Java logging frameworks is available here. The code example is available at GitHub in the Spring-Logging-Dependencies directory. Spring uses Jakarta Commons Logging API (JCL). Unfortunately, many people do not like its runtime discovery algorithm. We can disactivate it and use SLF4J with Logback instead. We will use a variation of the Spring MVC with Annotations example to do so. Here is the modified controller:   import org.springframework.stereotype.Controller; import org.springframework.ui.Model; import org.springframework.web.bind.annotation.RequestMapping;@Controller public class MyController {private static final Logger LOG = LoggerFactory.getLogger(MyController.class);@RequestMapping(value = '/') public String home(Model model) {String s = 'Logging at: ' + System.currentTimeMillis(); LOG.info(s);model.addAttribute('LogMsg', s);return 'index';}} We create a SFL4J logger and log some information with current time in milliseconds. The maven dependencies are: <properties> ... <spring.version>3.1.2.RELEASE</spring.version> <slf4j.version>1.7.1</slf4j.version> <logback.version>0.9.30</logback.version> </properties><dependency> <groupId>org.springframework</groupId> <artifactId>spring-context</artifactId> <version>${spring.version}</version> <exclusions> <exclusion> <groupId>commons-logging</groupId> <artifactId>commons-logging</artifactId> </exclusion> </exclusions> <type>jar</type> </dependency><dependency> <groupId>org.slf4j</groupId> <artifactId>jcl-over-slf4j</artifactId> <version>${slf4j.version}</version> <scope>runtime</scope> </dependency><dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>${slf4j.version}</version> <type>jar</type> </dependency><dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>${logback.version}</version> </dependency> Once built, one can start the example by browsing: http://localhost:9393/spring-logging-dependencies/. It will display the following:In the logs, you will find the logged statement:More Spring posts here.   Reference: Setting Logging Dependencies In Spring from our JCG partner Jerome Versrynge at the Technical Notes blog. ...
java-logo

Prototype Design Pattern: Creating another dolly

It’s really a time consuming process to create objects and also an expensive affair. So we are now on a venture to save both time and money. How do we do that?Anybody remember about Dolly? Yes, it’s the sheep which was the first mammal to be cloned. Well I don’t want to dig into the details but the key point is it’s all about cloning. It’s about creating a duplicate.Prototype Design Pattern is pretty much similar to this real life example. This is another in the series of Creational Design Pattern part of the Gang of Four Design Pattern. So this pattern works by cloning of an object rather than creation unlike Factory patterns. When to use this pattern?If the cost of creating the object is expensive or complicated. When trying to keep the number of classes in an application to a minimum When adding or removing objects at runtime When the client application needs to be unaware of the object creation, composition and representation. Objects are required which are similar to the existing objectsWhat does Prototype pattern do? Prototype pattern allows making new instances by copying the existing instances. Prototype pattern results in a cloned object which is different from the original object. The state of the original is the same as the clone, at the time of cloning. Thereafter each object may undergo state change. We can modify the objects to perform different things as well. The only good thing is client can make new instances without knowing which specific class is being instantiated. Structure: The prototype class declares an interface for cloning itself by implementing the Cloneable interface and using the clone() method. Concrete Prototype implements the clone() method for cloning itself. And the client class creates a new object by asking Prototype to clone itself rather than using the new keyword.The flow of events works in such a manner that the original class (e.g. Class A) is already initialized and instantiated. This is because we cannot use the clone as it is. We need to instantiate the original class (Class A) before using it. The client then requests the Prototype class for a new object of the same type as Class A. A concrete prototype depending on the type of object needed provides the object by cloning itself using the clone() method. Imagine a scenario where there might be requirements where we have to get the user profile data from the backend for multiple processing e.g. user profile or roles etc which does not get changed very frequently. So we might have to use expensive database resources, connections and transactions. In this case we can store the data in single call and cache it in the session for further processing. In the above example UserProfile object is the main object which will be cloned. The UserProfile implements the Cloneable interface. The BankDetails and Identity classes inherit from the UserProfile class. These are the concrete prototype classes. We have introduced a new class called UserProfileRegistry which finds appropriate UserProfile instance and then returns the clone to the client class appropriately.You would need to clone() an Object when you want to create another Object at runtime that is a true copy of the Object you are cloning. True copy means all the attributes of the newly created Object should be the same as the Object you are cloning. If you could have instantiated the class by using new instead, you would get an Object with all attributes as their initial values. For example, if you are designing a system for performing bank account transactions, then you would want to make a copy of the Object that holds your account information, perform transactions on it, and then replace the original Object with the modified one. In such cases, you would want to use clone() instead of new.   Interesting points:The Creational Design pattern can co-exist together e.g. Abstract factory, Builder and Prototype can use Singleton Pattern during their implementation or they might also work in isolation. Prototype pattern definitely need initialize operation but don’t need sub classing but Factory Method requires subclassing but don’t require initialize operation. Beneficial in Bank based transaction where we have expensive database queries. Caching might help and prototype pattern is the best answer to this situation as copy of the object having bank account information or user profile information can be used , perform transaction on it and then replace the original object with the modified one. The above example uses Shallow cloning method. However we can implement through deep cloning as well. A detailed explanation on this topic can be found in our article: Deep diving into Cloning.Benefits:Hides complexities of creating of objects. The clients can get new objects without knowing whose type it will be. Reduce subclassing.Drawback:Drawback to using the Prototype is that making a copy of an object can sometimes be complicated. Classes that have circular references to other classes cannot really be cloned.Download Source Code:  Reference: Prototype Design Pattern: Creating another dolly from our JCG partner Mainak Goswami at the Idiotechie blog. ...
java-logo

Java 7: File Filtering using NIO.2 – Part 2

Hello all. This is Part 2 of the File Filtering using NIO.2 series. For those of you who haven’t read Part 1, here’s a recap. NIO.2 is a new API for I/O operations included in the JDK since Java 7. With this new API, you can perform the same operations performed with java.io plus a lot of great functionalities such as: Accessing file metadata and watching for directory changes, among others. Obviously, the java.io package is not going to disappear because of backward compatibility, but we are encouraged to start using NIO.2 for our new I/O requirements. In this post, we are going to see how easy it is to filter the contents of a directory using this API. There are 3 ways in order to do so, we already review one way in Part 1 and now we are going to see another approach. What you need NetBeans 7+ or any other IDE that supports Java 7 JDK 7+ Filtering content of a directory is a common task in some applications and NIO.2 makes it really easy. The classes and Interfaces we are going to use are described next:java.nio.file.Path: Interface whose objects may represent files or directories in a file system. It’s like the java.io.File but in NIO.2. Whatever I/O operation you want to perform, you need an instance of this interface. java.nio.file.PathMatcher: Interface that allows objects to perform match operations on paths. java.nio.file.DirectoryStream: Interface whose objects iterate over the content of a directory. java.nio.file.Files: Class with static methods that operates on files, directories, etc.The way we are going to filter the contents of a directory is by using objects that implement the java.nio.file.PathMatcher interface. We can get one of these objects with the help of the java.nio.file.Files class, using the method +getPathMatcher(String):PathMatcher. This method supports both ‘glob’ and ‘regex’ patterns. You can check Part 1 of File Filtering using NIO.2 for more information about ‘glob’ and for ‘regex’ visit the java.util.regex.Pattern class. The pattern is matched against the name of the files, directories, etc. That live inside the directory. This is important to remember, using this method you can only filter by the name of the file, directory, etc. For example, if you want to filter .png and .jpg images, you should use one of the following syntax and pattern (notice the colon between the syntax and the pattern):‘glob:*.{png,jpg}’ ‘regex:([^\s]+(\.(?i)(png|jpg))$)’Of course, ‘glob’ syntax is much simpler, but you have the option of using regular expressions for a more detailed match. Anyway, you may be wondering why you should use this approach if the java.nio.files.DirectoryStream interface allows you to filter directly using ‘glob’… Well, let’s suppose that you already have a filter, but you need to perform more than one filtering operation, that’s when you need to use this approach. The following piece of code defines a method which scans a directory using different patterns: //in a class.../** * Scans the directory using the patterns passed * as parameters. * Only 3 patterns will be used. * @param folder directory to scan * @param patterns The first pattern will be used * as the glob pattern for the DirectoryStream. */ private static void scan(String folder, String... patterns) { //obtains the Images directory in the app directory Path dir = Paths.get(folder); //the Files class offers methods for validation if (!Files.exists(dir) || !Files.isDirectory(dir)) { System.out.println('No such directory!'); return; } //validate at least the glob pattern if (patterns == null || patterns.length < 1) { System.out.println( 'Please provide at least the glob pattern.'); return; }//obtain the objects that implements PathMatcher PathMatcher extraFilterOne = null; PathMatcher extraFilterTwo = null; if (patterns.length > 1 && patterns[1] != null) { extraFilterOne = FileSystems.getDefault(). getPathMatcher(patterns[1]); } if (patterns.length > 2 && patterns[2] != null) { extraFilterTwo = FileSystems.getDefault(). getPathMatcher(patterns[2]); }//Try with resources... so nice! try (DirectoryStreamds = Files.newDirectoryStream(dir, patterns[0])) { //iterate over the content of the directory and apply //any other extra pattern int count = 0; for (Path path : ds) { System.out.println( 'Evaluating ' + path.getFileName());if (extraFilterOne != null && extraFilterOne.matches(path.getFileName())) { System.out.println( 'Match found Do something!'); }if (extraFilterTwo != null && extraFilterTwo.matches(path.getFileName())) { System.out.println( 'Match found Do something else!'); }count++; } System.out.println(); System.out.printf( '%d Files match the global pattern\n', count); } catch (IOException ex) { ex.printStackTrace(); } } You can try invoking the last method with the following parameters:C:\Images or /Images depending on your OS. ?_*.jpg This pattern specifies that you want all .jpg images whose name starts with one digit followed by an underscore. glob:2_* Specifies another filter (using glob syntax) where you only want items whose name starts with the number two followed by an underscore. glob:3_* Specifies another filter (using glob syntax) where you only want items whose name starts with the number three followed by an underscore.Having several filters allows you to take different actions for matched items. Following is the result of the execution on my windows machine:And on my Linux virtual machine:Again, Write once, run everywhere! However, notice that the ordering of the items is system dependent, so do not ever hardcode the position of a file or directory. I hope you enjoyed this post, there is another more powerful way to filter the content of a directory and we’ll explore it in Part 3. Click here to download the source code.   Reference: Java 7: File Filtering using NIO.2 – Part 2 from our JCG partner Alexis Lopez at the Java and ME blog. ...
java-logo

Applying a Namespace During JAXB Unmarshal

For some an XML schema is a strict set of rules for how the XML document must be structured. But for others it is a general guideline to indicate what the XML should look like. This means that sometimes people want to accept input that doesn’t conform to the XML schema for some reason. In this example I will demonstrate how this can be done by leveraging a SAX XMLFilter. Java Model Below is the Java model that will be used for this example. Customer   package blog.namespace.sax;import javax.xml.bind.annotation.XmlRootElement;@XmlRootElement public class Customer {private String name;public String getName() { return name; }public void setName(String name) { this.name = name; }}   package-info We will use the package level @XmlSchema annotation to specify the namespace qualification for our model. @XmlSchema( namespace='http://www.example.com/customer', elementFormDefault=XmlNsForm.QUALIFIED) package blog.namespace.sax;import javax.xml.bind.annotation.*;   XML Input (input.xml) Even though our metadata specified that all the elements should be qualified with a namespace (http://www.example.com/customer) our input document is not namespace qualified. An XMLFilter will be used to add the namespace during the unmarshal operation. <?xml version='1.0' encoding='UTF-8'?> <customer> <name>Jane Doe</name> </customer>   XMLFilter (NamespaceFilter) The easiest way to create an XMLFilter is to extend XMLFilterImpl. For our use case we will override the startElement and endElement methods. In each of these methods we will call the corresponding super method passing in the default namespace as the URI parameter. package blog.namespace.sax;import org.xml.sax.*; import org.xml.sax.helpers.XMLFilterImpl;public class NamespaceFilter extends XMLFilterImpl {private static final String NAMESPACE = 'http://www.example.com/customer';@Override public void endElement(String uri, String localName, String qName) throws SAXException { super.endElement(NAMESPACE, localName, qName); }@Override public void startElement(String uri, String localName, String qName, Attributes atts) throws SAXException { super.startElement(NAMESPACE, localName, qName, atts); }}   Demo In the demo code below we will do a SAX parse of the XML document. The XMLReader will be wrapped in our XMLFilter. We will leverage JAXB’s UnmarshallerHandler as the ContentHandler. Once the parse has been done we can ask the UnmarshallerHandler for the resulting Customer object. package blog.namespace.sax;import javax.xml.bind.*; import javax.xml.parsers.*; import org.xml.sax.*;public class Demo {public static void main(String[] args) throws Exception { // Create the JAXBContext JAXBContext jc = JAXBContext.newInstance(Customer.class);// Create the XMLFilter XMLFilter filter = new NamespaceFilter();// Set the parent XMLReader on the XMLFilter SAXParserFactory spf = SAXParserFactory.newInstance(); SAXParser sp = spf.newSAXParser(); XMLReader xr = sp.getXMLReader(); filter.setParent(xr);// Set UnmarshallerHandler as ContentHandler on XMLFilter Unmarshaller unmarshaller = jc.createUnmarshaller(); UnmarshallerHandler unmarshallerHandler = unmarshaller .getUnmarshallerHandler(); filter.setContentHandler(unmarshallerHandler);// Parse the XML InputSource xml = new InputSource('src/blog/namespace/sax/input.xml'); filter.parse(xml); Customer customer = (Customer) unmarshallerHandler.getResult();// Marshal the Customer object back to XML Marshaller marshaller = jc.createMarshaller(); marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true); marshaller.marshal(customer, System.out); }}   Output Below is the output from running the demo code. Note how the output contains the namespace qualification based on the metadata. <?xml version='1.0' encoding='UTF-8'?> <customer xmlns='http://www.example.com/customer'> <name>Jane Doe</name> </customer>   Further Reading If you enjoyed this post then you may also be interested in:JAXB & Namespaces Preventing Entity Expansion Attacks in JAXB  Reference: Applying a Namespace During JAXB Unmarshal from our JCG partner Blaise Doughan at the Java XML & JSON Binding blog. ...
grails-logo

Grails SQL Logging part 2 – groovy.sql.Sql

I discussed options for logging Hibernate-generated SQL in an earlier post but today I was trying to figure out how to see the SQL from groovy.sql.Sql and didn’t have much luck at first. The core problem is that the Sql class uses a java.util.logging.Logger (JUL) while the rest of the world uses a Log4j logger (often with a Commons Logging or SLF4J wrapper). I assumed that since I am using the Grails support for JUL -> Log4j bridging (enabled with the grails.logging.jul.usebridge = true setting in Config.groovy) that all I needed to do was add the class name to my log4j DSL block:         log4j = { error 'org.codehaus.groovy.grails', 'org.springframework', 'org.hibernate', 'net.sf.ehcache.hibernate' debug 'groovy.sql.Sql' } but that didn’t work. Some googling led to this mailing list discussion which has a solution involving a custom java.util.logging.Handler to pipe JUL log messages for the 'groovy.sql.Sql' logger to Log4j. That seemed like overkill to me since theoretically that’s exactly what grails.logging.jul.usebridge = true already does. I realized I had no idea how the bridging worked, so I started looking at the implementation of this feature. It turns out that this is handled by the Grails “logging” plugin (org.codehaus.groovy.grails.plugins.log4j.LoggingGrailsPlugin) which calls org.slf4j.bridge.SLF4JBridgeHandler.install(). This essentially registers a listener that receives every JUL log message and pipes it to the corresponding SLF4J logger (typically wrapping a Log4j logger) with a sensible mapping of the different log levels (e.g. FINEST -> TRACE, FINER -> DEBUG, etc.) So what’s the problem then? While grails.logging.jul.usebridge = true does configure message routing, it doesn’t apply level settings from the log4j block to the corresponding JUL loggers. So although I set the level of 'groovy.sql.Sql' to debug, the JUL logger level is still at the default level (INFO). So all I need to do is programmatically set the logger’s level to DEBUG (or TRACE to see everything) once, e.g. in BootStrap.groovy import groovy.sql.Sql import java.util.logging.Levelclass BootStrap {def init = { servletContext -> Sql.LOG.level = Level.FINE } }   Reference: Grails SQL Logging part 2 – groovy.sql.Sql from our JCG partner Burt Beckwith at the An Army of Solipsists blog. ...
apache-hadoop-logo

Testing Hadoop Programs with MRUnit

 This post will take a slight detour from implementing the patterns found in Data-Intensive Processing with MapReduce to discuss something equally important, testing. I was inspired in part from a presentation by Tom Wheeler that I attended while at the 2012 Strata/Hadoop World conference in New York. When working with large data sets, unit testing might not be the first thing that comes to mind. However, when you consider the fact that no matter how large your cluster is, or how much data you have, the same code is pushed out to all nodes for running the MapReduce job, Hadoop mappers and reducers lend themselves very well to being unit tested. But what is not easy about unit testing Hadoop, is the framework itself. Luckily there is a library that makes testing Hadoop fairly easy – MRUnit. MRUnit is based on JUnit and allows for the unit testing of mappers, reducers and some limited integration testing of the mapper – reducer interaction along with combiners, custom counters and partitioners. We are using the latest release of MRUnit as of this writing, 0.9.0. All of the code under test comes from the previous post on computing averages using local aggregation. Setup To get started, download MRUnit from here. After you have extracted the tar file, cd into the mrunit-0.9.0-incubating/lib directory. In there you should see the following:mrunit-0.9.0-incubating-hadoop1.jar mrunit-0.9.0-incubating-hadoop2.jarAs I’m sure can guess, the mrunit-0.9.0-incubating-hadoop1.jar is for MapReduce version 1 of Hadoop and mrunit-0.9.0-incubating-hadoop2.jar is for working the new version of Hadoop’s MapReduce. For this post, and all others going forward, we will be using hadoop-2.0 version from Cloudera’s CDH4.1.1 release so we will need the mrunit-0.9.0-incubating-hadoop2.jar file. I added MRUnit, JUnit and Mockito as libraries in Intellij (JUnit and Mockito are found in the same directory as the MRUnit jar files). Now that we have set up our dependencies, let’s start testing. Testing Mappers Setting up to test a mapper is very straight forward and is best explained by looking at some code first. We will use the in-mapper combining example from the previous post: @Test public void testCombiningMapper() throws Exception { new MapDriver<LongWritable,Text,Text,TemperatureAveragingPair>() .withMapper(new AverageTemperatureCombiningMapper()) .withInput(new LongWritable(4),new Text(temps[3])) .withOutput(new Text('190101'),new TemperatureAveragingPair(-61,1)) .runTest(); } Notice the fluent api style which adds the ease of creating the test. To write your test you would:Instantiate an instance of the MapDriver class parameterized exactly as the mapper under test. Add an instance of the Mapper you are testing in the withMapper call. In the withInput call pass in your key and input value, in this case a LongWritable with an arbitrary value and a Text object that contains a line from from the NCDC weather dataset contained in a String array called ‘temps’ that was set up earlier in the test (not displayed here as it would take away from the presentation). Specify the expected output in the withOutput call, here we are expecting a Text object with the value of “190101″ and a TemperatureAveragingPair object containing the values -61 (temperature) and a 1 (count). The last call runTest feeds the specified input values into the mapper and compares the actual output against the expected output set in the ‘withOutput’ method.One thing to note is the MapDriver only allows one input and output per test. You can call withInput and withOutput multiple times if you want, but the MapDriver will overwrite the existing values with the new ones, so you will only ever be testing with one input/output at any time. To specify multiple inputs we would use the MapReduceDriver, covered a couple of sections later, but next up is testing the reducer. Testing Reducers Testing the reducer follows the same pattern as the mapper test. Again, let’s start by looking at a code example: @Test public void testReducerCold(){ List<TemperatureAveragingPair> pairList = new ArrayList<TemperatureAveragingPair>(); pairList.add(new TemperatureAveragingPair(-78,1)); pairList.add(new TemperatureAveragingPair(-84,1)); pairList.add(new TemperatureAveragingPair(-28,1)); pairList.add(new TemperatureAveragingPair(-56,1));new ReduceDriver<Text,TemperatureAveragingPair,Text,IntWritable>() .withReducer(new AverageTemperatureReducer()) .withInput(new Text('190101'), pairList) .withOutput(new Text('190101'),new IntWritable(-61)) .runTest(); }The test starts by creating a list of TemperatureAveragingPair objects to be used as the input to the reducer. A ReducerDriver is instantiated, and like the MapperDriver, is parameterized exactly as the reducer under test. Next we pass in an instance of the reducer we want to test in the withReducer call. In the withInput call we pass in the key of “190101″ and the pairList object created at the start of the test. Next we specify the output that we expect our reducer to emit, the same key of “190101″ and an IntWritable representing the average of the temperatures in the list. Finally runTest is called, which feeds our reducer the inputs specified and compares the output from the reducer against the expect output.The ReducerDriver has the same limitation as the MapperDriver of not accepting more than one input/output pair. So far we have tested the Mapper and Reducer in isolation, but we would also like to test them together in an integration test. Integration testing can be accomplished by using the MapReduceDriver class. The MapReduceDriver is also the class to use for testing the use of combiners, custom counters or custom partitioners. Integration Testing To test your mapper and reducer working together, MRUnit provides the MapReduceDriver class. The MapReduceDriver class as you would expect by now, with 2 main differences. First, you parameterize the input and output types of the mapper and the input and output types of the reducer. Since the mapper output types need to match the reducer input types, you end up with 3 pairs of parameterized types. Secondly you can provide multiple inputs and specify multiple expected outputs. Here is our sample code: @Test public void testMapReduce(){new MapReduceDriver<LongWritable,Text, Text,TemperatureAveragingPair, Text,IntWritable>() .withMapper(new AverageTemperatureMapper()) .withInput(new LongWritable(1),new Text(temps[0])) .withInput(new LongWritable(2),new Text(temps[1])) .withInput(new LongWritable(3),new Text(temps[2])) .withInput(new LongWritable(4),new Text(temps[3])) .withInput(new LongWritable(5),new Text(temps[6])) .withInput(new LongWritable(6),new Text(temps[7])) .withInput(new LongWritable(7),new Text(temps[8])) .withInput(new LongWritable(8),new Text(temps[9])) .withCombiner(new AverageTemperatureCombiner()) .withReducer(new AverageTemperatureReducer()) .withOutput(new Text('190101'),new IntWritable(-22)) .withOutput(new Text('190102'),new IntWritable(-40)) .runTest(); } As you can see from the example above, the setup is the same as the MapDriver and the ReduceDriver classes. You pass in instances of the mapper, reducer and optionally a combiner to test. The MapReduceDriver allows us to pass in multiple inputs that have different keys. Here the ‘temps’ array is the same one referenced in the mapper sample and contains a few lines from the NCDC weather dataset and the keys in those sample lines are the months of January and February of the year 1901 represented as “190101″ and “190102″ respectively. This test is successful, so we gain a little more confidence around the correctness of our mapper and reducer working together. Conclusion Hopefully, we have made the case for how useful MRUnit can be for testing Hadoop programs. I’d like to wrap this post up with some of my own observations. Although MRUnit makes unit testing easy for mapper and reducer code, the mapper and reducer examples presented here are fairly simple. If your map and/or reduce code starts to become more complex, it’s probably a better to decouple the code from the Hadoop framework and test the new classes on their own. Also, as useful as the MapReduceDriver class is for integration testing, it’s very easy to get to a point where you are no longer testing your code, but the Hadoop framework itself, which has already been done. I’ve come up with my own testing strategy that I intend to use going forward:Unit test the map/reduce code. Possibly write one integration test with the MapReduceDriver class. As a sanity check, run a MapReduce job on a single node install (on my laptop) to ensure it runs on the Hadoop framework. Then run my code on a test cluster, on EC2 using Apache Whirr in my case.Covering how to set up a single node install on my laptop (OSX Lion) and standing up a cluster on EC2 using Whirr would make this post too long, so I’ll cover those topics in the next one. Thanks for your time. ResourcesData-Intensive Processing with MapReduce by Jimmy Lin and Chris Dyer Hadoop: The Definitive Guide by Tom White Source Code from blog Hadoop API MRUnit for unit testing Apache Hadoop map reduce jobs Project Gutenberg a great source of books in plain text format, great for testing Hadoop jobs locally.  Reference: Testing Hadoop Programs with MRUnit from our JCG partner Bill Bejeck at the Random Thoughts On Coding blog. ...
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below:
Close