Apache Server and JMeter debugging

I had been using JMeter to generate load for my production server to test my application. The test-plan has 13+ HTTP sampler to make different request and one Regular Expression extractor to extract some value from the response. This value was used in consecutive HTTP Sampler. This test case simple and straight forward. Initially I used 200 JMeter threads to simulate 200 users. Server was able to handle these many requests easily but when the number of thread was increased, it couldn’t handled and waited infinitely. Surely something was going on. JMeter threads were waiting for the connection and couldn’t get it so waited infinitely. To avoid this situation, I introduced “HTTP Request default” to add some connection and response timeout. One problem was solved, now threads are not hanging there infinitely but they were timing out with the following exception.

java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
	at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
	at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
	at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
	at org.apache.jmeter.protocol.http.sampler.HTTPHC3Impl.sample(HTTPHC3Impl.java:258)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1088)
	at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1077)
	at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:428)
	at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
	at java.lang.Thread.run(Thread.java:662)

Definitely some-part of server (Apache 2.2.14 + Apache-tomcat-7.0.11) was creaking but not sure which part. Surely there was some bottleneck in the setup. In the current setup Apache server was forwarding requests to Tomcat engine. So surely either one of them were not able to handle 200+ request at a time. I changed the setup a little bit to forward all request directly to Tomcat engine. It was able to handle it, that means Apache was slacking. I quickly checked the Apache error.log file present at /var/log/apache2/error.log and found the following lines.

[Wed Jun 26 16:46:19 2013] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting
[Wed Jun 26 16:46:20 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Wed Jun 26 17:24:42 2013] [error] server is within MinSpareThreads of MaxClients, consider raising the MaxClients setting
[Wed Jun 26 17:24:43 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting

This clearly indicated that the MaxClients number should be increase which will in-turn increase the number of Apache thread. I quickly edited the apache2.conf file to increase the MaxClients number by putting the following configuration and executed "apache2ctl -t" to confirm that the changes made in configuration files were correct.

<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25  
    MaxSpareThreads     100 
    ThreadLimit         800
    ThreadsPerChild     800 
    MaxClients         2400
    MaxRequestsPerChild   0   
</IfModule>

I was bit relaxed thinking that now Apache will definitely handle the load now. But out of my surprise, it couldn’t and again the same exception hit the JMeter again. At this point I wanted to check the Apache performance by enabling the server-status. This feature can give you clear idea about the connection states. I put the following lines in the /etc/apache2/mods-available/status.conf

<Location /server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from  .your.domain.here.com localhost
</Location>

I restarted Apache and server by executing the following command

/> sudo service apache2 restart; sudo service tomcat restart;

Once the Apache restarted I hit the following URL to get the server-status.

http://your.server.name/server-status?refresh=3

After running the JMeter for 200+ threads I noticed this on the status page.

Apache server status

The status page indicated that there were lots of connection in "W" (Sending Reply) state. This is can be caused by other reasons. I tried to google it but couldn’t find any definite solution. But one thing was sure that the Apache is not causing problem. It must be between Apache and tomcat. I realized that JMeter test plan works fine for 200 threads and even Tomcat by default has 200 threads. I just took a chance and increased the number of threads to 400 by editing "<APACHE_TOMCAT_HOME>/conf/server.xml" file.

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" maxThreads="400" minSpareThreads="20" />

Now Server can handle the JMeter load easily and Apache connections are in good state even after finishing the test-plan.
Although problem was fixed, I couldn’t understand why Tomcat was able to handle the load when JMeter was configured to send all requests directly to Tomcat but couldn’t handle the load when Apache was forwarding the request to Tomcat. If you have any explanation about this behavior please share below in comment section.
 

Reference: Apache Server and JMeter debugging from our JCG partner Rakesh Cusat at the Code4Reference blog.

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

JPA Mini Book

Learn how to leverage the power of JPA in order to create robust and flexible Java applications. With this Mini Book, you will get introduced to JPA and smoothly transition to more advanced concepts.

JVM Troubleshooting Guide

The Java virtual machine is really the foundation of any Java EE platform. Learn how to master it with this advanced guide!

Given email address is already subscribed, thank you!
Oops. Something went wrong. Please try again later.
Please provide a valid email address.
Thank you, your sign-up request was successful! Please check your e-mail inbox.
Please complete the CAPTCHA.
Please fill in the required fields.

2 Responses to "Apache Server and JMeter debugging"

  1. Vel says:

    Rakesh,

    I have faced the same issue, when hitting tomcat through jmeter threads routed via a Apache webserver. Tomcat goes down when the total no. of active threads exceeds configured limit of threads in tomcat. Are you sure your issue by increasing the tomcat maxlimit and by-passing the apache webserver?

  2. @vel
    Yes, I am sure it did work for me. You can try yourself and let me know your findings.

Leave a Reply


3 × = six



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below:
Close