About Pierre Hugues Charbonneau

Pierre-Hugues Charbonneau (nickname P-H) is working for CGI Inc. Canada for the last 10 years as a senior IT consultant. His primary area of expertise is Java EE, middleware & JVM technologies. He is a specialist in production system troubleshooting, root cause analysis, middleware, JVM tuning, scalability and capacity improvement; including internal processes improvement for IT support teams. P-H is the principal author at Java EE Support Patterns.

JVM: How to analyze Thread Dump

This article will teach you how to analyze a JVM Thread Dump and pinpoint the root cause of your problem(s). From my perspective, Thread Dump analysis is the most important skillset to master for any individual involved in Java EE production support. The amount of information that you can derive from Thread Dump snapshots is often much beyond than what you can think of.

My goal is to share with you my knowledge on Thread Dump analysis that I accumulated over the last 10 years e.g. hundreds of Thread Dump analysis cycles with dozens of common problem patterns across many JVM versions and JVM vendors.

Please bookmark this page and stay tuned for weekly articles.
Please also feel free to share this Thread Dump training plan with your work colleagues and friends.

Sounds good, I really need to improve my Thread Dump skills… so where do we start?

What I’m proposing to you is a complete Thread Dump training plan. The following items will be covered. I will also provide you with real life Thread Dump examples that you can study and understand.

1)  Thread Dump overview & fundamentals
2)  Thread Dump generation techniques and available tools
3)  Thread Dump format differences between Sun HotSpot, IBM JRE and Oracle JRockit
4)  Thread Stack Trace explanation and interpretation
5)  Thread Dump analysis and correlation techniques
6)  Thread Dump common problem patterns (Thread race, deadlock, hanging IO calls, garbage collection / OutOfMemoryError problems, infinite looping etc.)
7)  Thread Dump examples via real life case studies

I really hope this Thread Dump analysis training plan will be beneficial for you so please stay tuned for weekly updates and articles!

But what if I still have questions or still struggling to understand these training articles?

Don’t worry and please consider me as your trainer. I strongly encourage you to ask me any question on Thread Dump (remember, there are no stupid questions) so I propose the following options to you for free; simply chose the communication model that you are more comfortable with:

1)  Submit your Thread Dump related question(s) by posting your comment(s) below the article (please feel free to remain Anonymous)
2)  Submit your Thread Dump data to the Root Cause Analysis forum
3)  Email me your Thread Dump related question(s) @phcharbonneau@hotmail.com

Can I send you my Thread Dump data from my production environment / servers?

Yes, please feel free to send me your generated Thread Dump data via email or Root Cause Analysis forum if you wish to discuss the root cause of your problem(s). Real life Thread Dump analysis is always the best way to learn.

I really hope that you will enjoy and share this Thread Dump analysis training plan. I will do my very best to provide you with quality material and answers to any question.

Before going deeper into Thread Dump analysis and problem patterns, it is very important that you understand the fundamentals. The post will cover the basics and allow you to better your JVM and middleware interaction with your Java EE container.

Java VM overview

The Java virtual machine is really the foundation of any Java EE platform. This is where your middleware and applications are deployed and active.

The JVM provides the middleware software and your Java / Java EE program with:

– A runtime environment for your Java / Java EE program (bytecode format)
– Several program features and utilities (IO facilities, data structure, Threads management, security, monitoring etc.)
– Dynamic memory allocation and management via the garbage collector

Your JVM can reside on many OS (Solaris, AIX, Windows etc.) and depending of your physical server specifications, you can install 1…n JVM processes per physical / virtual server.

JVM and Middleware software interactions

Find below a diagram showing you a high level interaction view between the JVM, middleware and application(s).

This is showing you a typical and simple interaction diagram between the JVM, middleware and application. As you can see, the Threads allocation for a standard Java EE application are done mainly between the middleware kernel itself and JVM (there are some exceptions when application itself or some APIs create Threads directly but this is not common and must be done very carefully).

Also, please note that certain Threads are managed internally within the JVM itself such as GC (garbage collection) Threads in order to handle concurrent garbage collections.

Since most of the Thread allocations are done by the Java EE container, it is important that you understand and recognize the Thread Stack Trace and identify it properly from the Thread Dump data. This will allow you to understand quickly the type of request that the Java EE container is attempting to execute.

From a Thread Dump analysis perspective, you will learn how to differentiate between the different Thread Pools found from the JVM and identify the request type.

This last section will provide you with an overview of what is a JVM Thread Dump for the HotSpot VM and the different Threads that you will find. Detail for the IBM VM Thread Dump format will be provided in the part 4.

Please note that you will find the Thread Dump sample used for this article from the root cause analysis forum.

JVM Thread Dump – what is it?

A JVM Thread Dump is a snapshot taken at a given time which provides you with a complete listing of all created Java Threads.

Each individual Java Thread found gives you information such as:

Thread name; often used by middleware vendors to identify the Thread Id along with its associated Thread Pool name and state (running, stuck etc.)

      – Thread type & priority ex: daemon prio=3 ** middleware softwares typically create their Threads as daemon meaning their Threads are running in background; providing services to its user e.g. your Java EE application **

       – Java Thread ID ex: tid=0x000000011e52a800 ** This is the Java Thread Id obtained via java.lang.Thread.getId() and usually implemented as an auto-incrementing long 1..n**

       – Native Thread ID ex: nid=0x251c** Crucial information as this native Thread Id allows you to correlate for example which Threads from an OS perspective are using the most CPU within your JVM etc. **

       – Java Thread State and detail ex: waiting for monitor entry [0xfffffffea5afb000] java.lang.Thread.State: BLOCKED (on object monitor)
** Allows to quickly learn about Thread state and its potential current blocking condition **

        – Java Thread Stack Trace; this is by far the most important data that you will find from the Thread Dump. This is also where you will spent most of your analysis time since the Java Stack Trace provides you with 90% of the information that you need in order to pinpoint root cause of many problem pattern types as you will learn later in the training sessions

        – Java Heap breakdown; starting with HotSpot VM 1.6, you will also find at the bottom of the Thread Dump snapshot a breakdown of the HotSpot memory spaces utilization such as your Java Heap (YoungGen, OldGen) & PermGen space. This is quite useful when excessive GC is suspected as a possible root cause so you can do out-of-the-box correlation with Thread data / patterns found

Heap
PSYoungGen      total 466944K, used 178734K [0xffffffff45c00000, 0xffffffff70800000, 0xffffffff70800000)
eden space 233472K, 76% used [0xffffffff45c00000,0xffffffff50ab7c50,0xffffffff54000000)
from space 233472K, 0% used [0xffffffff62400000,0xffffffff62400000,0xffffffff70800000)
to   space 233472K, 0% used [0xffffffff54000000,0xffffffff54000000,0xffffffff62400000)
PSOldGen        total 1400832K, used 1400831K [0xfffffffef0400000, 0xffffffff45c00000, 0xffffffff45c00000)
object space 1400832K, 99% used [0xfffffffef0400000,0xffffffff45bfffb8,0xffffffff45c00000)
PSPermGen       total 262144K, used 248475K [0xfffffffed0400000, 0xfffffffee0400000, 0xfffffffef0400000)
object space 262144K, 94% used [0xfffffffed0400000,0xfffffffedf6a6f08,0xfffffffee0400000)

Thread Dump breakdown overview

In order for you to better understand, find below a diagram showing you a visual breakdown of a HotSpot VM Thread Dump and its common Thread Pools found:

As you can there are several pieces of information that you can find from a HotSpot VM Thread Dump. Some of these pieces will be more important than others depending of your problem pattern(problem patterns will be simulated and explained in future articles).

For now, find below a detailed explanation for each Thread Dump section as per our sample HotSpot Thread Dump:

# Full thread dump identifier
This is basically the unique keyword that you will find in your middleware / standalong Java standard output log once you generate a Thread Dump (ex: via kill -3 <PID> for UNIX). This is the beginning of the Thread Dump snapshot data.

Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.0-b11 mixed mode):

# Java EE middleware, third party & custom application Threads
This portion is the core of the Thread Dump and where you will typically spend most of your analysis time. The number of Threads found will depend on your middleware software that you use, third party libraries (that might have its own Threads) and your application (if creating any custom Thread, which is generally not a best practice).

In our sample Thread Dump, Weblogic is the middleware used. Starting with Weblogic 9.2, a self-tuning Thread Pool is used with unique identifier “’weblogic.kernel.Default (self-tuning)”

"[STANDBY] ExecuteThread: '414' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x000000010916a800 nid=0x2613 in Object.wait() [0xfffffffe9edff000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0xffffffff27d44de0> (a weblogic.work.ExecuteThread)
        at java.lang.Object.wait(Object.java:485)
        at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:160)
        - locked <0xffffffff27d44de0> (a weblogic.work.ExecuteThread)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

# HotSpot VM Thread
This is an internal Thread managed by the HotSpot VM in order to perform internal native operations. Typically you should not worry about this one unless you see high CPU(via Thread Dump & prstat / native Thread id correlation).

"VM Periodic Task Thread" prio=3 tid=0x0000000101238800 nid=0x19 waiting on condition

# HotSpot GC Thread
When using HotSpot parallel GC (quite common these days when using multi physical cores hardware), the HotSpot VM create by default or as per your JVM tuning a certain # of GC Threads. These GC Threads allow the VM to perform its periodic GC cleanups in a parallel manner, leading to an overall reduction of the GC time; at the expense of increased CPU utilization.

"GC task thread#0 (ParallelGC)" prio=3 tid=0x0000000100120000 nid=0x3 runnable
"GC task thread#1 (ParallelGC)" prio=3 tid=0x0000000100131000 nid=0x4 runnable
………………………………………………………………………………………………………………………………………………………………

This is crucial data as well since when facing GC related problems such as excessive GC, memory leaks etc, you will be able to correlate any high CPU observed from the OS / Java process(es) with these Threads using their native id value (nid=0x3). You will learn how to identify and confirm this problem is future articles.

# JNI global references count
JNI (Java Native Interface) global references are basically Object references from the native code to a Java object managed by the Java garbage collector. Its role is to prevent collection of an object that is still in use by native code but technically with no “live” references in the Java code.

It is also important to keep an eye on JNI references in order to detect JNI related leaks. This can happen if you program use JNI directly or using third party tools like monitoring tools which are prone to native memory leaks.

JNI global references: 1925

# Java Heap utilization view
This data was added back to JDK 1 .6 and provides you with a short and fast view of your HotSpot Heap. I find it quite useful when troubleshooting GC related problems along with HIGH CPU since you get both Thread Dump & Java Heap in a single snapshot allowing you to determine (or to rule out) any pressure point in a particular Java Heap memory space along with current Thread computing currently being done at that time. As you can see in our sample Thread Dump, the Java Heap OldGen is maxed out!

Heap
 PSYoungGen      total 466944K, used 178734K [0xffffffff45c00000, 0xffffffff70800000, 0xffffffff70800000)
  eden space 233472K, 76% used [0xffffffff45c00000,0xffffffff50ab7c50,0xffffffff54000000)
  from space 233472K, 0% used [0xffffffff62400000,0xffffffff62400000,0xffffffff70800000)
  to   space 233472K, 0% used [0xffffffff54000000,0xffffffff54000000,0xffffffff62400000)
 PSOldGen        total 1400832K, used 1400831K [0xfffffffef0400000, 0xffffffff45c00000, 0xffffffff45c00000)
  object space 1400832K, 99% used [0xfffffffef0400000,0xffffffff45bfffb8,0xffffffff45c00000)
 PSPermGen       total 262144K, used 248475K [0xfffffffed0400000, 0xfffffffee0400000, 0xfffffffef0400000)
  object space 262144K, 94% used [0xfffffffed0400000,0xfffffffedf6a6f08,0xfffffffee0400000)

I hope this article has helped to understand the basic view of a HotSpot VM Thread Dump.The next article will provide you this same Thread Dump overview and breakdown for the IBM VM.

Please feel free to post any comment or question.

Reference: How to analyze Thread Dump – part 1 How to analyze Thread Dump – Part2: JVM Overview & How to analyze Thread Dump – Part 3: HotSpot VM from our JCG partner Pierre-Hugues Charbonneau at the Java EE Support Patterns & Java Tutorial blog.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

JPA Mini Book

Learn how to leverage the power of JPA in order to create robust and flexible Java applications. With this Mini Book, you will get introduced to JPA and smoothly transition to more advanced concepts.

JVM Troubleshooting Guide

The Java virtual machine is really the foundation of any Java EE platform. Learn how to master it with this advanced guide!

Given email address is already subscribed, thank you!
Oops. Something went wrong. Please try again later.
Please provide a valid email address.
Thank you, your sign-up request was successful! Please check your e-mail inbox.
Please complete the CAPTCHA.
Please fill in the required fields.

10 Responses to "JVM: How to analyze Thread Dump"

  1. Hoàng Hưng says:

    thanks

    • Pierre-Hugues Charbonneau says:

      Hi,

       

      Please note that I will release the next series shortly
      which will include Thread Dump Stack Trace analysis approach and common problem
      patterns that will be available to share on Java Code Geeks. Thread Dump analysis complexity
      is often due to so many different problem patterns so this should really help any Java
      / Java EE individual involved in either application support or development (load testing
      etc.) to identify such patterns more quickly. This series will also include data correlation
      techniques; especially for problems related to high CPU, excessive garbage
      collection etc.

       

      I’m also looking forward for any feedback and “wish list” of what
      aspect of Thread Dump analysis you want more detail on.

       

      Regards,

      Pierre-Hugues

  2. This “JVM: How to analyze Thread Dump” article of your is ridiculously similar to “How to Analyze Java Thread Dumps” http://www.cubrid.org/blog/dev-platform/how-to-analyze-java-thread-dumps/ published couple months ago by CUBRID developers on their official blog.
    I strongly believe these two should go together hand in hand as they mutually supportive. Great article, by the way!

  3. C Jacob says:

    My application is running in tomcat.
    If the server is running for many days without restarting tomcat, we face a problem as the server CPU usage will go above 100% very frequently and come back to normal immediately. By the time, we take thread dump, it come back to normal. However on continuous observation it is found that the high cpu usage nid is pointing to a line in thread dump

    “Concurrent Mark-Sweep GC Thread” prio=10 tid=0x00007fb3581e3800 nid=0x74f3 runnable

    Few more lines are given below

    “VM Thread” prio=10 tid=0x00007fb35826e800 nid=0x74f4 runnable

    “Gang worker#0 (Parallel GC Threads)” prio=10 tid=0x00007fb358012800 nid=0x74d7 runnable

    “Gang worker#1 (Parallel GC Threads)” prio=10 tid=0x00007fb358014000 nid=0x74d8 runnable

    “Gang worker#2 (Parallel GC Threads)” prio=10 tid=0x00007fb358016000 nid=0x74d9 runnable

    “Gang worker#3 (Parallel GC Threads)” prio=10 tid=0x00007fb358018000 nid=0x74da runnable

    “Gang worker#4 (Parallel GC Threads)” prio=10 tid=0x00007fb358019800 nid=0x74dc runnable

    “Gang worker#5 (Parallel GC Threads)” prio=10 tid=0x00007fb35801b800 nid=0x74dd runnable

    “Gang worker#6 (Parallel GC Threads)” prio=10 tid=0x00007fb35801d800 nid=0x74de runnable

    “Gang worker#7 (Parallel GC Threads)” prio=10 tid=0x00007fb35801f000 nid=0x74df runnable

    “Gang worker#8 (Parallel GC Threads)” prio=10 tid=0x00007fb358021000 nid=0x74e0 runnable

    “Gang worker#9 (Parallel GC Threads)” prio=10 tid=0x00007fb358023000 nid=0x74e1 runnable

    “Gang worker#10 (Parallel GC Threads)” prio=10 tid=0x00007fb358024800 nid=0x74e2 runnable

    “Gang worker#11 (Parallel GC Threads)” prio=10 tid=0x00007fb358026800 nid=0x74e3 runnable

    “Gang worker#12 (Parallel GC Threads)” prio=10 tid=0x00007fb358028800 nid=0x74e4 runnable

    “Gang worker#13 (Parallel GC Threads)” prio=10 tid=0x00007fb35802a000 nid=0x74e5 runnable

    “Gang worker#14 (Parallel GC Threads)” prio=10 tid=0x00007fb35802c000 nid=0x74e6 runnable

    “Gang worker#15 (Parallel GC Threads)” prio=10 tid=0x00007fb35802e000 nid=0x74e7 runnable

    “Concurrent Mark-Sweep GC Thread” prio=10 tid=0x00007fb3581e3800 nid=0x74f3 runnable
    “Gang worker#0 (Parallel CMS Threads)” prio=10 tid=0x00007fb3581df000 nid=0x74f1 runnable

    “Gang worker#1 (Parallel CMS Threads)” prio=10 tid=0x00007fb3581e1000 nid=0x74f2 runnable

    “VM Periodic Task Thread” prio=10 tid=0x00007fb35849c800 nid=0x7501 waiting on condition

    JNI global references: 6075

    Any help….

    • Hi C Jacob,

      It is likely that your problem is related to a major GC collection. You have quite a few GC threads created, I would recommend to verify if the increased CPU is due to GC threads performing excessive garbage collection, typical during a major collection. With 16 GC threads concurrently , this can significantly impact CPU utilization until the memory is fully reclaimed.

      The CMS collector has the tendency to fragment over time, at some point requiring a major collection which can trigger quite a large surge of your CPU utilization.

      One more recommendation, enable -verbose:gc or use JVisualVM to monitor your heap utilization. This will help you correlate the GC process with your CPU utilization spikes.

      Regards,
      P-H

  4. Wallace Roberto says:

    HI, thanks very much.

    I’ve started work with weblogic support , And i have some problems to identify this problems wich can be discovered analyzing thread dump

    att

  5. James says:

    Why do you write that creating custom threads is not best practice?

  6. Hank says:

    System was built on
    J2EE/Struts
    Tomcat Catalina
    IIS

    Get frequently crash(Tomcat) problem. even after input enough RAM recently.

  7. Neel says:

    Task Logs: ‘attempt_201407291140_0007_m_000002_0′

    stdout logs

    2014-07-29 11:52:26
    Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode):

    “IPC Client (339343228) connection to HDN003/192.168.101.233:60020 from root” daemon prio=10 tid=0x00007f7ef91fe800 nid=0x316c runnable [0x00007f7ee0191000]
    java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    – locked (a sun.nio.ch.Util$2)
    – locked (a java.util.Collections$UnmodifiableSet)
    – locked (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at org.apache.hadoop.hbase.ipc.RpcClient$Connection$PingInputStream.read(RpcClient.java:555)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
    – locked (a java.io.BufferedInputStream)
    at java.io.DataInputStream.readInt(DataInputStream.java:387)
    at org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1059)
    at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:721)

    “org.apache.hadoop.hdfs.PeerCache@42d795d2″ daemon prio=10 tid=0x00007f7ef9264000 nid=0x202d waiting on condition [0x00007f7ee0313000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
    at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
    at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
    at java.lang.Thread.run(Thread.java:744)

    “main-EventThread” daemon prio=10 tid=0x00007f7ef8e5d000 nid=0x201a waiting on condition [0x00007f7ee0515000]
    java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    – parking to wait for (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

    “main-SendThread(HDN002:2181)” daemon prio=10 tid=0x00007f7ef8e3b000 nid=0x2019 runnable [0x00007f7ee0616000]
    java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    – locked (a sun.nio.ch.Util$2)
    – locked (a java.util.Collections$UnmodifiableSet)
    – locked (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1075)

    “communication thread” daemon prio=10 tid=0x00007f7ef8da8000 nid=0x2017 in Object.wait() [0x00007f7ee0717000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.lang.Object)
    at org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:659)
    – locked (a java.lang.Object)
    at java.lang.Thread.run(Thread.java:744)

    “Thread-7″ daemon prio=10 tid=0x00007f7ef8d74800 nid=0x2016 runnable [0x00007f7ee0818000]
    java.lang.Thread.State: RUNNABLE
    at org.apache.hadoop.net.unix.DomainSocketWatcher.doPoll0(Native Method)
    at org.apache.hadoop.net.unix.DomainSocketWatcher.access$800(DomainSocketWatcher.java:52)
    at org.apache.hadoop.net.unix.DomainSocketWatcher$1.run(DomainSocketWatcher.java:457)
    at java.lang.Thread.run(Thread.java:744)

    “Timer thread for monitoring jvm” daemon prio=10 tid=0x00007f7ef8c73000 nid=0x2015 in Object.wait() [0x00007f7ee0919000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:552)
    – locked (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:505)

    “IPC Parameter Sending Thread #0″ daemon prio=10 tid=0x00007f7ef8c27000 nid=0x2014 waiting on condition [0x00007f7ee0a25000]
    java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    – parking to wait for (a java.util.concurrent.SynchronousQueue$TransferStack)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
    at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

    “IPC Client (339343228) connection to /127.0.0.1:48629 from job_201407291140_0007″ daemon prio=10 tid=0x00007f7ef8c23800 nid=0x2013 in Object.wait() [0x00007f7ee0b26000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a org.apache.hadoop.ipc.Client$Connection)
    at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:902)
    – locked (a org.apache.hadoop.ipc.Client$Connection)
    at org.apache.hadoop.ipc.Client$Connection.run(Client.java:947)

    “Thread for syncLogs” daemon prio=10 tid=0x00007f7ef8aca000 nid=0x200c waiting on condition [0x00007f7ee1073000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.hadoop.mapred.Child$3.run(Child.java:156)

    “Service Thread” daemon prio=10 tid=0x00007f7ef80c0000 nid=0x1ffe runnable [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    “C2 CompilerThread1″ daemon prio=10 tid=0x00007f7ef80bd800 nid=0x1ffd waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    “C2 CompilerThread0″ daemon prio=10 tid=0x00007f7ef80ba800 nid=0x1ffc waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    “Signal Dispatcher” daemon prio=10 tid=0x00007f7ef80b0800 nid=0x1ffb waiting on condition [0x0000000000000000]
    java.lang.Thread.State: RUNNABLE

    “Finalizer” daemon prio=10 tid=0x00007f7ef8099800 nid=0x1ffa in Object.wait() [0x00007f7eec776000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
    – locked (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)

    “Reference Handler” daemon prio=10 tid=0x00007f7ef8095800 nid=0x1ff9 in Object.wait() [0x00007f7eec877000]
    java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:503)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
    – locked (a java.lang.ref.Reference$Lock)

    “main” prio=10 tid=0x00007f7ef8015000 nid=0x1fe5 in Object.wait() [0x00007f7effd87000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    – waiting on (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
    at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1435)
    – locked (a org.apache.hadoop.hbase.ipc.RpcClient$Call)
    at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1653)
    at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1711)
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:27332)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:168)
    at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:120)
    – locked (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:96)
    – locked (a org.apache.hadoop.hbase.client.RpcRetryingCaller)
    at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:336)
    at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:194)
    at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:138)
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483)
    at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)

    “VM Thread” prio=10 tid=0x00007f7ef8093000 nid=0x1ff8 runnable

    “GC task thread#0 (ParallelGC)” prio=10 tid=0x00007f7ef802b000 nid=0x1fe6 runnable

    “GC task thread#1 (ParallelGC)” prio=10 tid=0x00007f7ef802d000 nid=0x1fe7 runnable

    “GC task thread#2 (ParallelGC)” prio=10 tid=0x00007f7ef802e800 nid=0x1fe8 runnable

    “GC task thread#3 (ParallelGC)” prio=10 tid=0x00007f7ef8030800 nid=0x1fe9 runnable

    “GC task thread#4 (ParallelGC)” prio=10 tid=0x00007f7ef8032800 nid=0x1fea runnable

    “GC task thread#5 (ParallelGC)” prio=10 tid=0x00007f7ef8034000 nid=0x1feb runnable

    “GC task thread#6 (ParallelGC)” prio=10 tid=0x00007f7ef8036000 nid=0x1fec runnable

    “GC task thread#7 (ParallelGC)” prio=10 tid=0x00007f7ef8038000 nid=0x1fed runnable

    “GC task thread#8 (ParallelGC)” prio=10 tid=0x00007f7ef803a000 nid=0x1fee runnable

    “GC task thread#9 (ParallelGC)” prio=10 tid=0x00007f7ef803b800 nid=0x1fef runnable

    “GC task thread#10 (ParallelGC)” prio=10 tid=0x00007f7ef803d800 nid=0x1ff0 runnable

    “GC task thread#11 (ParallelGC)” prio=10 tid=0x00007f7ef803f800 nid=0x1ff1 runnable

    “GC task thread#12 (ParallelGC)” prio=10 tid=0x00007f7ef8041000 nid=0x1ff2 runnable

    “GC task thread#13 (ParallelGC)” prio=10 tid=0x00007f7ef8043000 nid=0x1ff3 runnable

    “GC task thread#14 (ParallelGC)” prio=10 tid=0x00007f7ef8045000 nid=0x1ff4 runnable

    “GC task thread#15 (ParallelGC)” prio=10 tid=0x00007f7ef8046800 nid=0x1ff5 runnable

    “GC task thread#16 (ParallelGC)” prio=10 tid=0x00007f7ef8048800 nid=0x1ff6 runnable

    “GC task thread#17 (ParallelGC)” prio=10 tid=0x00007f7ef804a800 nid=0x1ff7 runnable

    “VM Periodic Task Thread” prio=10 tid=0x00007f7ef80d3000 nid=0x1fff waiting on condition

    JNI global references: 234

    Heap
    PSYoungGen total 225792K, used 139775K [0x00000000eaa80000, 0x00000000fa600000, 0x0000000100000000)
    eden space 194048K, 64% used [0x00000000eaa80000,0x00000000f2463210,0x00000000f6800000)
    from space 31744K, 47% used [0x00000000f6800000,0x00000000f769ca68,0x00000000f8700000)
    to space 31744K, 0% used [0x00000000f8700000,0x00000000f8700000,0x00000000fa600000)
    ParOldGen total 514560K, used 80K [0x00000000c0000000, 0x00000000df680000, 0x00000000eaa80000)
    object space 514560K, 0% used [0x00000000c0000000,0x00000000c0014020,0x00000000df680000)
    PSPermGen total 27648K, used 27402K [0x00000000bae00000, 0x00000000bc900000, 0x00000000c0000000)
    object space 27648K, 99% used [0x00000000bae00000,0x00000000bc8c28d0,0x00000000bc900000)

    stderr logs

    syslog logs

    2014-07-29 11:42:27,714 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
    2014-07-29 11:42:27,788 WARN org.apache.hadoop.conf.Configuration: /data/disk1/mapred/local/taskTracker/root/jobcache/job_201407291140_0007/job.xml:an attempt to override final parameter: mapred.child.java.opts; Ignoring.
    2014-07-29 11:42:28,184 INFO org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
    2014-07-29 11:42:28,186 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId=
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.0.2–1, built on 06/09/2014 16:09 GMT
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=HDN003
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.7.0_45
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_45-cloudera/jre
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/var/run/cloudera-scm-agent/process/6239-mapreduce-TASKTRACKER:/usr/java/jdk1.7.0_45-cloudera/lib/tools.jar:/opt/cloudera/parcels/2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/opt/cloudera/parcels/HADOOP_LZO-0.4.15-1.gplextras.p0.33/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-5.0.2-1.cdh5.0.2.p0.13/lib/hadoop-0.20-mapreduce/lib/native/Linux-amd64-64:/data/disk4/mapred/local/taskTracker/root/jobcache/job_201407291140_0007/attempt_201407291140_0007_m_000002_0/work
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/data/disk4/mapred/local/taskTracker/root/jobcache/job_201407291140_0007/attempt_201407291140_0007_m_000002_0/work/tmp
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.20.3.el6.x86_64
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=mapred
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hadoop-mapreduce
    2014-07-29 11:42:28,760 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/data/disk4/mapred/local/taskTracker/root/jobcache/job_201407291140_0007/attempt_201407291140_0007_m_000002_0/work
    2014-07-29 11:42:28,761 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=HDN003:2181,HNN001:2181,HDN002:2181,HDN001:2181 sessionTimeout=1200000 watcher=hconnection-0x2f8a3e5f, quorum=HDN003:2181,HNN001:2181,HDN002:2181,HDN001:2181, baseZNode=/hbase
    2014-07-29 11:42:28,772 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2f8a3e5f connecting to ZooKeeper ensemble=HDN003:2181,HNN001:2181,HDN002:2181,HDN001:2181
    2014-07-29 11:42:28,776 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server HDN002/192.168.101.232:2181. Will not attempt to authenticate using SASL (unknown error)
    2014-07-29 11:42:28,776 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to HDN002/192.168.101.232:2181, initiating session
    2014-07-29 11:42:28,803 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server HDN002/192.168.101.232:2181, sessionid = 0x24780c328c001c9, negotiated timeout = 1200000
    2014-07-29 11:42:29,309 WARN org.apache.hadoop.hbase.HBaseConfiguration: Config option “hbase.regionserver.lease.period” is deprecated. Instead, use “hbase.client.scanner.timeout.period”
    2014-07-29 11:42:29,333 INFO org.apache.hadoop.hbase.mapreduce.TableOutputFormat: Created table instance for TomTailortouchpoints
    2014-07-29 11:42:29,341 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
    2014-07-29 11:42:29,345 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4a21efa3
    2014-07-29 11:42:29,974 INFO org.apache.hadoop.mapred.MapTask: Processing split: HDN003:888_77_E4AD502EF5579022617E96E7C92FAA9B_20140202133243417_29170354,999_999999999999*
    2014-07-29 11:42:29,978 WARN org.apache.hadoop.hbase.HBaseConfiguration: Config option “hbase.regionserver.lease.period” is deprecated. Instead, use “hbase.client.scanner.timeout.period”
    2014-07-29 11:42:29,985 WARN org.apache.hadoop.hbase.HBaseConfiguration: Config option “hbase.regionserver.lease.period” is deprecated. Instead, use “hbase.client.scanner.timeout.period”

Leave a Reply


five + = 14



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below:
Close