JMX and Spring – Part 3

This article is the last one of this series. Take a look at Part 1 and Part 2.

In this last article of the series I’ll show how to use the native JMX support within the JDK to implement a notification mechanism which alerts a listener when the HEAP memory is above a certain threshold.

As discussed in my previous article this approach is ideal because is push instead of pull, is not intrusive and places minimal computing demand on your application.

These are the key components to the solution illustrated in this article:

  • MemoryWarningService: This component acts as a listener and registers itself with the Memory MBean to receive notifications. It is configurable with a threshold in the form of a percentage between 0 and 1 (where 1 is 100%)
  • MemoryThreadDumper: This component is invoked when the MemoryWarningService is notified that the HEAP usage is above the threshold and its responsibility is to write a thread dump to a file
  • MemoryWarningServiceConfigurator: This component is an MBean and exposes a method to change the threshold of the MemoryWarningService.

The solution provides also a MemoryHeapFiller class used to fill up the HEAP while testing the application and a MemTest class to bootstrap the Spring environment.

While the application is running (play with the MemoryHeapFiller settings) You can fire the JConsole at URL: service:jmx:rmi://localhost/jndi/rmi://localhost:8888/jemosJmxConnector connecting as jemosAdmin / secure and change the threshold to various values.

The code is not meant for production: it is not robust, there are numerous comments missing, and the filename where to write the thread dump is hard-coded; it represents, however, a good starting point.

The code is attached below. You will need Maven to build it.

Download Jemos-jmx-experiments-0.0.1-SNAPSHOT-project

I tried a scenario with initial threshold to be 0.5, I changed it to 0.3 and then to 0.8. The results are shown below:

2011-08-15 21:53:21 ClassPathXmlApplicationContext [INFO] Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@a4a63d8: startup date [Mon Aug 15 21:53:21 BST 2011]; root of context hierarchy
2011-08-15 21:53:21 XmlBeanDefinitionReader [INFO] Loading XML bean definitions from class path resource [jemos-jmx-appCtx.xml]
2011-08-15 21:53:21 PropertyPlaceholderConfigurer [INFO] Loading properties file from class path resource [jemos-jmx.properties]
2011-08-15 21:53:21 PropertyPlaceholderConfigurer [INFO] Loading properties file from URL [file:/C:/Users/mtedone/.secure/jmxconnector-credentials.properties]
2011-08-15 21:53:21 ThreadPoolTaskScheduler [INFO] Initializing ExecutorService  'myScheduler'
2011-08-15 21:53:21 ClassPathXmlApplicationContext [INFO] Bean 'myScheduler' of type [class org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2011-08-15 21:53:21 DefaultListableBeanFactory [INFO] Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@30296f76: defining beans [propertyConfigurer,loggerConfigurator,memoryWarningServiceConfigurator,memoryHeapFiller,memoryThreadDumper,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,mbeanExporter,jemosJmxServer,rmiRegistry,clientConnector,memoryMxBean,memoryWarningService,org.springframework.scheduling.annotation.internalAsyncAnnotationProcessor,org.springframework.scheduling.annotation.internalScheduledAnnotationProcessor,myScheduler]; root of factory hierarchy
2011-08-15 21:53:21 AnnotationMBeanExporter [INFO] Registering beans for JMX exposure on startup
2011-08-15 21:53:21 RmiRegistryFactoryBean [INFO] Looking for RMI registry at port '8888'
2011-08-15 21:53:23 RmiRegistryFactoryBean [INFO] Could not detect RMI registry - creating new one
2011-08-15 21:53:23 ConnectorServerFactoryBean [INFO] JMX connector server started: javax.management.remote.rmi.RMIConnectorServer@4355d3a3
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Bean with name 'jemosJmxServer' has been autodetected for JMX exposure
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Bean with name 'loggerConfigurator' has been autodetected for JMX exposure
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Bean with name 'memoryWarningServiceConfigurator' has been autodetected for JMX exposure
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Located managed bean 'loggerConfigurator': registering with JMX server as MBean [jemos.mbeans:type=config,name=LoggingConfiguration]
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Located MBean 'jemosJmxServer': registering with JMX server as MBean [jemos.mbeans:name=jemosJmxServer,type=RMIConnectorServer]
2011-08-15 21:53:23 AnnotationMBeanExporter [INFO] Located managed bean 'memoryWarningServiceConfigurator': registering with JMX server as MBean [jemos.mbeans:type=config,name=MemoryWarningServiceConfiguration]
2011-08-15 21:53:23 MemoryWarningService [INFO] Percentage is: 0.5
2011-08-15 21:53:23 MemoryWarningService [INFO] Listener added to JMX bean
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
2011-08-15 21:53:37 MemoryWarningService [INFO] Percentage is: 0.3
2011-08-15 21:53:37 MemoryWarningServiceConfigurator [INFO] Memory threshold set to 0.3
Adding data...
2011-08-15 21:53:38 MemoryWarningService [WARN] Memory usage low!!!
2011-08-15 21:53:38 MemoryWarningService [WARN] percentageUsed = 0.3815679398794023
2011-08-15 21:53:38 MemoryThreadDumper [WARN] Stacks dumped to: C:/tmp/stacks.dump
Adding data...
Adding data...
Adding data...
2011-08-15 21:53:45 MemoryWarningService [INFO] Percentage is: 0.8
2011-08-15 21:53:45 MemoryWarningServiceConfigurator [INFO] Memory threshold set to 0.8
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
Adding data...
2011-08-15 21:54:01 MemoryWarningService [WARN] Memory usage low!!!
2011-08-15 21:54:01 MemoryWarningService [WARN] percentageUsed = 0.8383333266727508
2011-08-15 21:54:02 MemoryThreadDumper [WARN] Stacks dumped to: C:/tmp/stacks.dump
Adding data...
Adding data...
Adding data...
Exception in thread "JMX server connection timeout 24" java.lang.OutOfMemoryError: Java heap space

The Memory Warning Service

package uk.co.jemos.experiments.jmx;

import java.lang.management.ManagementFactory;
import java.lang.management.MemoryNotificationInfo;
import java.lang.management.MemoryPoolMXBean;
import java.lang.management.MemoryType;

import javax.annotation.PostConstruct;
import javax.management.Notification;
import javax.management.NotificationEmitter;
import javax.management.NotificationListener;

import org.springframework.beans.factory.annotation.Autowired;

/**
 * A component which sends notifications when the HEAP memory is above a certain
 * threshold.
 * 
 * @author mtedone
 * 
 */
public class MemoryWarningService implements NotificationListener {

    /** This bean's name */
    public static final String MBEAN_NAME = "jemos.mbeans:type=monitoring,name=MemoryWarningService";

    /** The application logger */
    private static final org.apache.log4j.Logger LOG = org.apache.log4j.Logger
            .getLogger(MemoryWarningService.class);

    @Autowired
    private NotificationEmitter memoryMxBean;

    @Autowired
    private MemoryThreadDumper threadDumper;

    /** A pool of Memory MX Beans specialised in HEAP management */
    private static final MemoryPoolMXBean tenuredGenPool = findTenuredGenPool();

    /**
     * {@inheritDoc}
     */
    @Override
    public void handleNotification(Notification notification, Object handback) {

        if (notification.getType().equals(
                MemoryNotificationInfo.MEMORY_THRESHOLD_EXCEEDED)) {
            long maxMemory = tenuredGenPool.getUsage().getMax();
            long usedMemory = tenuredGenPool.getUsage().getUsed();
            LOG.warn("Memory usage low!!!");
            double percentageUsed = (double) usedMemory / maxMemory;
            LOG.warn("percentageUsed = " + percentageUsed);
            threadDumper.dumpStacks();

        } else {
            LOG.info("Other notification received..."
                    + notification.getMessage());
        }

    }

    /**
     * It sets the threshold percentage.
     * 
     * @param percentage
     */
    public void setPercentageUsageThreshold(double percentage) {
        if (percentage <= 0.0 || percentage > 1.0) {
            throw new IllegalArgumentException("Percentage not in range");
        } else {
            LOG.info("Percentage is: " + percentage);
        }
        long maxMemory = tenuredGenPool.getUsage().getMax();
        long warningThreshold = (long) (maxMemory * percentage);
        tenuredGenPool.setUsageThreshold(warningThreshold);
    }

    @PostConstruct
    public void completeSetup() {
        memoryMxBean.addNotificationListener(this, null, null);
        LOG.info("Listener added to JMX bean");
    }

    /**
     * Tenured Space Pool can be determined by it being of type HEAP and by it
     * being possible to set the usage threshold.
     */
    private static MemoryPoolMXBean findTenuredGenPool() {
        for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {
            // I don't know whether this approach is better, or whether
            // we should rather check for the pool name "Tenured Gen"?
            if (pool.getType() == MemoryType.HEAP
                    && pool.isUsageThresholdSupported()) {
                return pool;
            }
        }
        throw new AssertionError("Could not find tenured space");
    }

}

The Memory Thread Dumper

package uk.co.jemos.experiments.jmx;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
import java.lang.management.ManagementFactory;
import java.lang.management.ThreadInfo;
import java.lang.management.ThreadMXBean;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;

import org.apache.commons.io.IOUtils;
import org.springframework.stereotype.Component;

/**
 * This component dumps the thread stack to the file system.
 * 
 * @author mtedone
 * 
 */
@Component
public class MemoryThreadDumper {

    /** The application logger */
    private static final org.apache.log4j.Logger LOG = org.apache.log4j.Logger
            .getLogger(MemoryThreadDumper.class);

    /**
     * It dumps the Thread stacks
     * 
     * @throws IOException
     */
    public void dumpStacks() {

        // hard-coded: This needs to be changed to a property or something
        String stackFileName = "C:/tmp/stacks.dump";

        ThreadMXBean mxBean = ManagementFactory.getThreadMXBean();
        ThreadInfo[] threadInfos = mxBean.getThreadInfo(
                mxBean.getAllThreadIds(), 0);
        Map<Long, ThreadInfo> threadInfoMap = new HashMap<Long, ThreadInfo>();
        for (ThreadInfo threadInfo : threadInfos) {
            threadInfoMap.put(threadInfo.getThreadId(), threadInfo);
        }

        File dumpFile = new File(stackFileName);
        BufferedWriter writer = null;
        try {
            writer = new BufferedWriter(new FileWriter(dumpFile));
            this.dumpTraces(mxBean, threadInfoMap, writer);

            LOG.warn("Stacks dumped to: " + stackFileName);

        } catch (IOException e) {
            throw new IllegalStateException(
                    "An exception occurred while writing the thread dump");
        } finally {
            IOUtils.closeQuietly(writer);
        }

    }

    private void dumpTraces(ThreadMXBean mxBean,
            Map<Long, ThreadInfo> threadInfoMap, Writer writer)
            throws IOException {
        Map<Thread, StackTraceElement[]> stacks = Thread.getAllStackTraces();
        writer.write("Dump of "
                + stacks.size()
                + " thread at "
                + new SimpleDateFormat("yyyy/MM/dd HH:mm:ss z")
                        .format(new Date(System.currentTimeMillis())) + "\n\n");
        for (Map.Entry<Thread, StackTraceElement[]> entry : stacks.entrySet()) {
            Thread thread = entry.getKey();
            writer.write("\"" + thread.getName() + "\" prio="
                    + thread.getPriority() + " tid=" + thread.getId() + " "
                    + thread.getState() + " "
                    + (thread.isDaemon() ? "deamon" : "worker") + "\n");
            ThreadInfo threadInfo = threadInfoMap.get(thread.getId());
            if (threadInfo != null) {
                writer.write("    native=" + threadInfo.isInNative()
                        + ", suspended=" + threadInfo.isSuspended()
                        + ", block=" + threadInfo.getBlockedCount() + ", wait="
                        + threadInfo.getWaitedCount() + "\n");
                writer.write("    lock=" + threadInfo.getLockName()
                        + " owned by " + threadInfo.getLockOwnerName() + " ("
                        + threadInfo.getLockOwnerId() + "), cpu="
                        + mxBean.getThreadCpuTime(threadInfo.getThreadId())
                        / 1000000L + ", user="
                        + mxBean.getThreadUserTime(threadInfo.getThreadId())
                        / 1000000L + "\n");
            }
            for (StackTraceElement element : entry.getValue()) {
                writer.write("        ");
                writer.write(element.toString());
                writer.write("\n");
            }
            writer.write("\n");
        }
    }

}

The Memory Service Configuration MBean

package uk.co.jemos.experiments.jmx.mbeans;

import org.springframework.beans.BeansException;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.jmx.export.annotation.ManagedOperation;
import org.springframework.jmx.export.annotation.ManagedOperationParameter;
import org.springframework.jmx.export.annotation.ManagedOperationParameters;
import org.springframework.jmx.export.annotation.ManagedResource;
import org.springframework.stereotype.Component;

import uk.co.jemos.experiments.jmx.MemoryWarningService;

@Component
@ManagedResource(objectName = MemoryWarningServiceConfigurator.MBEAN_NAME, //
description = "Allows clients to set the memory threshold")
public class MemoryWarningServiceConfigurator implements
        ApplicationContextAware {

    

    /** The application logger */
    private static final org.apache.log4j.Logger LOG = org.apache.log4j.Logger
            .getLogger(MemoryWarningServiceConfigurator.class);

    public static final String MBEAN_NAME = "jemos.mbeans:type=config,name=MemoryWarningServiceConfiguration";

    private ApplicationContext ctx;

    @ManagedOperation(description = "Sets the memory threshold for the memory warning system")
    @ManagedOperationParameters({ @ManagedOperationParameter(description = "The memory threshold", name = "memoryThreshold"), })
    public void setMemoryThreshold(double memoryThreshold) {

        MemoryWarningService memoryWarningService = (MemoryWarningService) ctx
                .getBean("memoryWarningService");
        memoryWarningService.setPercentageUsageThreshold(memoryThreshold);

        LOG.info("Memory threshold set to " + memoryThreshold);
    }

    @Override
    public void setApplicationContext(ApplicationContext applicationContext)
            throws BeansException {
        ctx = applicationContext;

    }


}

The Spring configuration

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
    xmlns:util="http://www.springframework.org/schema/util"
    xmlns:task="http://www.springframework.org/schema/task"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
        http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.0.xsd
        http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd">


    <bean id="propertyConfigurer" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"  >
        <property name="locations">
            <list>
                <value>classpath:jemos-jmx.properties</value>
                <value>file:///${user.home}/.secure/jmxconnector-credentials.properties
                </value>
            </list>
        </property>
    </bean>

    <context:component-scan base-package="uk.co.jemos.experiments.jmx" />

    <context:mbean-export default-domain="jemos.mbeans" />

    <bean id="jemosJmxServer" class="org.springframework.jmx.support.ConnectorServerFactoryBean"       
        depends-on="rmiRegistry">
        <property name="objectName" value="connector:name=rmi" />
        <property name="serviceUrl"
            value="service:jmx:rmi://localhost/jndi/rmi://localhost:${jemos.jmx.rmi.port}/jemosJmxConnector" />
        <property name="environment">
            <!-- the following is only valid when the sun jmx implementation is used -->
            <map>
                <entry key="jmx.remote.x.password.file" value="${user.home}/.secure/jmxremote.password" />
                <entry key="jmx.remote.x.access.file" value="${user.home}/.secure/jmxremote.access" />
            </map>
        </property>
    </bean>

    <bean id="rmiRegistry" class="org.springframework.remoting.rmi.RmiRegistryFactoryBean">
        <property name="port" value="${jemos.jmx.rmi.port}" />
    </bean>

    <bean id="clientConnector" class="org.springframework.jmx.support.MBeanServerConnectionFactoryBean"       
        depends-on="jemosJmxServer">
        <property name="serviceUrl"
            value="service:jmx:rmi://localhost/jndi/rmi://localhost:${jemos.jmx.rmi.port}/jemosJmxConnector" />
        <property name="environment">
            <map>
                <entry key="jmx.remote.credentials">
                    <bean
                        factory-method="commaDelimitedListToStringArray">
                        <constructor-arg value="${jmx.username},${jmx.password}" />
                    </bean>
                </entry>
            </map>
        </property>
    </bean>


    <bean id="memoryMxBean" class="java.lang.management.ManagementFactory"
        factory-method="getMemoryMXBean" />
        
    <bean id="memoryWarningService" class="uk.co.jemos.experiments.jmx.MemoryWarningService">
      <property name="percentageUsageThreshold" value="0.5" />
    </bean>
        

    <task:annotation-driven scheduler="myScheduler" />   

    <task:scheduler id="myScheduler" pool-size="10" />    
        


</beans>

 

Reference: JMX and Spring – Part 3 from our JCG partner Marco Tedone at the Marco Tedone’s blog blog.

Related Whitepaper:

Functional Programming in Java: Harnessing the Power of Java 8 Lambda Expressions

Get ready to program in a whole new way!

Functional Programming in Java will help you quickly get on top of the new, essential Java 8 language features and the functional style that will change and improve your code. This short, targeted book will help you make the paradigm shift from the old imperative way to a less error-prone, more elegant, and concise coding style that’s also a breeze to parallelize. You’ll explore the syntax and semantics of lambda expressions, method and constructor references, and functional interfaces. You’ll design and write applications better using the new standards in Java 8 and the JDK.

Get it Now!  

Leave a Reply


+ 5 = seven



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books