About Jakub Holy

Jakub is an experienced Java[EE] developer working for a lean & agile consultancy in Norway. He is interested in code quality, developer productivity, testing, and in how to make projects succeed.

VisualVM: Monitoring Remote JVM Over SSH (JMX Or Not)

VisualVM is a great tool for monitoring JVM (5.0+) regarding memory usage, threads, GC, MBeans etc. Let’s see how to use it over SSH to monitor (or even profile, using its sampler) a remote JVM either with JMX or without it. This post is based on Sun JVM 1.6 running on Ubuntu 10 and VisualVM 1.3.3.

1. Communication: JStatD vs. JMX

There are two modes of communication between VisualVM and the JVM: Either over the Java Management Extensions (JMX) protocol or over jstatd.

jstatd

jstatd is a daemon that is distributed with JDK. You start it from the command line (it’s likely necessary to run it as the user running the target JVM or as root) on the target machine and VisualVM will contact it to fetch information about the remote JVMs.

  • Advantages: Can connect to a running JVM, no need to start it with special parameters
  • Disadvantages: Much more limited monitoring capabilities (f.ex. no CPU usage monitoring, not possible to run the Sampler and/or take thread dumps).

Ex.:

bash> cat jstatd.all.policy
grant codebase 'file:${java.home}/../lib/tools.jar' {
permission java.security.AllPermission;
}
bash> sudo /path/to/JDK/bin/jstatd -J-Djava.security.policy=jstatd.all.policy
# You can specify port with -p number and get more info with -J-Djava.rmi.server.logCalls=true

Note: Replace “${java.home}/../lib/tools.jar” with the absolute “/path/to/jdk/lib/tools.jar” if you have only copied but not installed the JDK.

If you get the failure

Could not create remote object
access denied (java.util.PropertyPermission java.rmi.server.ignoreSubClasses write)
java.security.AccessControlException: access denied (java.util.PropertyPermission java.rmi.server.ignoreSubClasses write)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:374)

then jstatd likely hasn’t been started with the right java.security.policy file (try to provide fully qualified path to it).

More info about VisualVM and jstatd from Oracle.

JMX

  • Advantages: Using JMX will give you the full power of VisualVM.
  • Disadvantages: Need to start the JVM with some system properties.

You will generally want to use something like the following properties when starting the target JVM (though you could also enable SSL and/or require username and password):

yourJavaCommand... -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=1098

See Remote JMX Connections.

2. Security: SSH

The easiest way to connect to the remote JMX or jstatd over ssh is to use a SOCKS proxy, which standard ssh clients can set up.

2.1 Set Up the SSH Tunnel With SOCKS

ssh -v -D 9696 my_server.example.com

2.2 Configure VisualVM to Use the Proxy

Tools->Options->Network – Manual Proxy Settings – check it and configure SOCKS Proxy at localhost and port 9696

2.3 Connect VisualVM to the Target

File -> Add Remote Host… – type the IP or hostname of the remote machine JStatD Connection

You should see logs both in the ssh window (thanks to its “-v”, f.ex. “debug1: Connection to port 9696 forwarding to socks port 0 requested.” and “debug1: channel 3: free: direct-tcpip: listening port 9696 for 10.2.47.71 port 1099, connect from 127.0.0.1 port 61262, nchannels 6“) and in the console where you started jstatd (many, f.ex. “FINER: RMI TCP Connection(23)-10.2.47.71: …“)

Wait few minutes after having added the remote host, you should then see the JVMs running there.

Available stats: JVM arguments, Monitor: Heap, classes, threads monitoring (but not CPU). Sampler and MBeans require JMX.

JMX

Right-click on the remote host you have added and select Add JMX Connection …, type the JMX port you have chosen.

You should see similar logs as with jstatd.

Available stats: Also CPU usage, system properties, detailed Threads report with access to stack traces, CPU sampling (memory sampling not supported).

Note: Sampler vs. Profiler

The VisualVM’s Sampler excludes time spent in Object.wait and Thread.sleep (f.ex. waiting on I/O). Use the NetBeans Profiler to profile or sample a remote application if you want to have more control or want the possibility to include Object.wait and Thread.sleep time. It requires its Remote Pack (a java agent, i.e. a JAR file) to be in the target JVM (NetBeans’ Attach Wizard can generate the remote pack for you in step 4, Manual integration, and show you the options to pass to the target JVM to use it).

You can run the profiler over SSH by forwarding its default port (5140) and attaching to the forwarded port at localhost.(NetBeans version 7.1.1.)

Don’t forget to share!

Reference: VisualVM: Monitoring Remote JVM Over SSH (JMX Or Not) from our JCG partner Jakub Holy at the The Holy Java blog.

Related Whitepaper:

Functional Programming in Java: Harnessing the Power of Java 8 Lambda Expressions

Get ready to program in a whole new way!

Functional Programming in Java will help you quickly get on top of the new, essential Java 8 language features and the functional style that will change and improve your code. This short, targeted book will help you make the paradigm shift from the old imperative way to a less error-prone, more elegant, and concise coding style that’s also a breeze to parallelize. You’ll explore the syntax and semantics of lambda expressions, method and constructor references, and functional interfaces. You’ll design and write applications better using the new standards in Java 8 and the JDK.

Get it Now!  

2 Responses to "VisualVM: Monitoring Remote JVM Over SSH (JMX Or Not)"

  1. Reiner Saddey says:

    Thanks Jakub!

    I tried in vain to visualvm into a firewalled (but sshable) JBoss using ssh single port redirection (ssh -L …).

    After successfully having verified your recommendations (“easiest way” using ssh -D to proxy) I’m now beginning to understand, how JVisualVM actually connects to the target VM when using JMX.

    1. JVisualVM connects to the remote JMX using the port configured -Dcom.sun.management.jmxremote.port

    2. Remote JMX returns a host name (as valid within the target system) and (arbitrary) port

    3. which in turn are then used by JVisualVM to connect to the remote VM.

    Thus, a single port redirection will not suffice to establish the connection to the target VM.

    Using a proxy effectively puts JVisualVM within the target system itself (as far as network connections are concerned) and thus allows it to connect to arbitrary ports and (and possibly names as well) within the target.

    Thanks again,

    Reiner

  2. Reiner Saddey says:

    Just an addendum for colleagues still struggling with Windows boxes :-)
    Putty can be used instead of ssh to implement the ad-hoc proxy magic. See http://www.virtualroadside.com/blog/index.php/2007/04/12/dynamic-socks-proxy-using-putty/

Leave a Reply


+ 8 = seventeen



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

15,153 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books