Apache Server and JMeter debugging

I had been using JMeter to generate load for my production server to test my application. The test-plan has 13+ HTTP sampler to make different request and one Regular Expression extractor to extract some value from the response. This value was used in consecutive HTTP Sampler. This test case simple and straight forward. Initially I used 200 JMeter threads to simulate 200 users. Server was able to handle these many requests easily but when the number of thread was increased, it couldn’t handled and waited infinitely. Surely something was going on. JMeter threads were waiting for the connection and couldn’t get it so waited infinitely. To avoid this situation, I introduced “HTTP Request default” to add some connection and response timeout. One problem was solved, now threads are not hanging there infinitely but they were timing out with the following exception.

java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
	at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
	at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
	at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
	at org.apache.jmeter.protocol.http.sampler.HTTPHC3Impl.sample(HTTPHC3Impl.java:258)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1088)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1077)
	at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:428)
	at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
	at java.lang.Thread.run(Thread.java:662)

Definitely some-part of server (Apache 2.2.14 + Apache-tomcat-7.0.11) was creaking but not sure which part. Surely there was some bottleneck in the setup. In the current setup Apache server was forwarding requests to Tomcat engine. So surely either one of them were not able to handle 200+ request at a time. I changed the setup a little bit to forward all request directly to Tomcat engine. It was able to handle it, that means Apache was slacking. I quickly checked the Apache error.log file present at /var/log/apache2/error.log and found the following lines.

[Wed Jun 26 16:46:19 2013] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting
[Wed Jun 26 16:46:20 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Wed Jun 26 17:24:42 2013] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting
[Wed Jun 26 17:24:43 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting

This clearly indicated that the MaxClients number should be increase which will in-turn increase the number of Apache thread. I quickly edited the apache2.conf file to increase the MaxClients number by putting the following configuration and executed "apache2ctl -t" to confirm that the changes made in configuration files were correct.

<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25  
    MaxSpareThreads     100 
    ThreadLimit         800
    ThreadsPerChild     800 
    MaxClients         2400
    MaxRequestsPerChild   0   
</IfModule>

I was bit relaxed thinking that now Apache will definitely handle the load now. But out of my surprise, it couldn’t and again the same exception hit the JMeter again. At this point I wanted to check the Apache performance by enabling the server-status. This feature can give you clear idea about the connection states. I put the following lines in the /etc/apache2/mods-available/status.conf

<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from  .your.domain.here.com localhost
</Location>

I restarted Apache and server by executing the following command

/> sudo service apache2 restart; sudo service tomcat restart;

Once the Apache restarted I hit the following URL to get the server-status.

http://your.server.name/server-status?refresh=3

After running the JMeter for 200+ threads I noticed this on the status page.

Apache server status

The status page indicated that there were lots of connection in "W" (Sending Reply) state. This is can be caused by other reasons. I tried to google it but couldn’t find any definite solution. But one thing was sure that the Apache is not causing problem. It must be between Apache and tomcat. I realized that JMeter test plan works fine for 200 threads and even Tomcat by default has 200 threads. I just took a chance and increased the number of threads to 400 by editing "<APACHE_TOMCAT_HOME>/conf/server.xml" file.

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="400" minSpareThreads="20" />

Now Server can handle the JMeter load easily and Apache connections are in good state even after finishing the test-plan.
Although problem was fixed, I couldn’t understand why Tomcat was able to handle the load when JMeter was configured to send all requests directly to Tomcat but couldn’t handle the load when Apache was forwarding the request to Tomcat. If you have any explanation about this behavior please share below in comment section.
 

Reference: Apache Server and JMeter debugging from our JCG partner Rakesh Cusat at the Code4Reference blog.
Related Whitepaper:

Functional Programming in Java: Harnessing the Power of Java 8 Lambda Expressions

Get ready to program in a whole new way!

Functional Programming in Java will help you quickly get on top of the new, essential Java 8 language features and the functional style that will change and improve your code. This short, targeted book will help you make the paradigm shift from the old imperative way to a less error-prone, more elegant, and concise coding style that’s also a breeze to parallelize. You’ll explore the syntax and semantics of lambda expressions, method and constructor references, and functional interfaces. You’ll design and write applications better using the new standards in Java 8 and the JDK.

Get it Now!  

2 Responses to "Apache Server and JMeter debugging"

  1. Vel says:

    Rakesh,

    I have faced the same issue, when hitting tomcat through jmeter threads routed via a Apache webserver. Tomcat goes down when the total no. of active threads exceeds configured limit of threads in tomcat. Are you sure your issue by increasing the tomcat maxlimit and by-passing the apache webserver?

  2. @vel
    Yes, I am sure it did work for me. You can try yourself and let me know your findings.

Leave a Reply


four + 6 =



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books