How to use Asynchronous Servlets to improve performance

This post is going to describe a performance optimization technique applicable to a common problem related to modern webapps. Applications nowadays are no longer just passively waiting for browsers to initiate requests, but want to start the communication themselves. A typical example could involve chat applications, auction houses, etc – the common denominator being the fact that most of the time the connections with the browser are idle and wait for a certain event being triggered.

This type of applications have developed a problem class of their own, especially when facing heavy load. The symptoms include starving threads, suffering user interaction, staleness issues, etc.

Based on recent experience with this type of apps under load, I thought it would be a good time to demonstrate a simple solution. After Servlet API 3.0 implementations became mainstream, the solution has become truly simple, standardized and elegant.

But before we jump into demonstrating the solution, we should understand the problem in more details. For our readers – what could be easier than to explain the problem with help of some source code:

@WebServlet(urlPatterns = "/BlockingServlet")
public class BlockingServlet extends HttpServlet {
  private static final long serialVersionUID = 1L;

  protected void doGet(HttpServletRequest request,
                       HttpServletResponse response) throws ServletException, IOException {
    try {
      long start = System.currentTimeMillis();
      Thread.sleep(2000);
      String name = Thread.currentThread().getName();
      long duration = System.currentTimeMillis() - start;
      response.getWriter().printf("Thread %s completed the task in %d ms.", name, duration);
    } catch (Exception e) {
      throw new RuntimeException(e.getMessage(), e);
    }
  }

The servlet above is an example of how an application described above could look like:

  • Request arrives, announcing interest to monitor certain events
  • The thread is blocked until the event arrives
  • Upon receiving the event, the response is compiled and sent back to the client

In order to keep it simple, we have replaced the waiting part with a call to Thread.sleep().

Now – this is a perfectly normal servlet you might think. In many cases, you are completely correct – there is nothing wrong with the code until the application faces significant load.

In order to simulate this load I created a fairly simple test with some help from JMeter, where I launched 2,000 threads, each running through 10 iterations of bombarding the application with requests to the /BlockedServlet. Running the test with the deployed servlet on an out-of-the-box Tomcat 7.0.42 I got the following results:

  • Average response time: 19,324 ms
  • Minimum response time: 2,000 ms
  • Maximum response time: 21,869 ms
  • Throughput: 97 requests/second

The default Tomcat configuration has got 200 worker threads which, coupled with the fact that the simulated work is replaced with 2,000ms sleep cycle explains nicely the minimum and maximum response times – in each second the 200 threads should be able to complete 100 sleep cycles, 2 seconds each. Adding context switch costs on top of this, the achieved throughput of 97 requests/second is pretty close to our expectations.

The throughput itself would not look too bad for 99.9% of the applications out there. Looking at the maximum and especially, average response times the problem starts to look more serious though. Getting the response in 20 seconds instead of expected 2 seconds is a sure way to annoy your users.

Let us now take a look at an alternative implementation, taking advantage of the Servlet API 3.0 asynchronous support:

@WebServlet(asyncSupported = true, value = "/AsyncServlet")
public class AsyncServlet extends HttpServlet {
  private static final long serialVersionUID = 1L;

  protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
    Work.add(request.startAsync());
  }
}
public class Work implements ServletContextListener {
  private static final BlockingQueue queue = new LinkedBlockingQueue();

  private volatile Thread thread;

  public static void add(AsyncContext c) {
    queue.add(c);
  }

  @Override
  public void contextInitialized(ServletContextEvent servletContextEvent) {
    thread = new Thread(new Runnable() {
      @Override
      public void run() {
        while (true) {
          try {
            Thread.sleep(2000);
            AsyncContext context;
            while ((context = queue.poll()) != null) {
              try {
                ServletResponse response = context.getResponse();
                response.setContentType("text/plain");
                PrintWriter out = response.getWriter();
                out.printf("Thread %s completed the task", Thread.currentThread().getName());
                out.flush();
              } catch (Exception e) {
                throw new RuntimeException(e.getMessage(), e);
              } finally {
                context.complete();
              }
            }
          } catch (InterruptedException e) {
            return;
          }
        }
      }
    });
    thread.start();
  }

  @Override
  public void contextDestroyed(ServletContextEvent servletContextEvent) {
    thread.interrupt();
  }
}

This bit of code is a bit more complex, so maybe before we start digging into solution details, I can outline that this solution performed ~75x better latency- and ~20x better throughput-wise. Equipped with knowledge of such results, you should be more motivated to understand what is actually going on in the second example.

The servlet itself looks truly simple. Two facts are worth outlining though, first of which declares the servlet to support asynchronous method invocations:

@WebServlet(asyncSupported = true, value = "/AsyncServlet")

Second important aspect is hidden in the following line

Work.add(request.startAsync());

in which the whole request processing is delegated to the Work class. The context of the request is stored using an AsyncContext instance holding the request and response that were provided by the container.

Now, the second and more complex class – the Work implemented as ServletContextListener starts looking simpler. Incoming requests are just queued in the implementation to wait for the notification – this could be an updated bid on the monitored auction or the next message in the group chat that all the requests are waiting for.

When the notification arrives – again, simplified as just waiting 2,000ms in a Thread.sleep(), all the blocked tasks in the queue are processed by a single worker thread responsible for compiling and sending the responses. Instead of blocking hundreds of threads to wait behind the external notification, we achieved this in a lot simpler and cleaner way – batching the interest groups together and processing the requests in a single thread.

And the results speak for themselves – the very same test on the very same Tomcat 7.0.24 with default configuration resulted in the following:

  • Average response time: 265 ms
  • Minimum response time: 6 ms
  • Maximum response time: 2,058 ms
  • Throughput: 1,965 requests/second

The specific case here is small and synthetic, but the similar improvements are achievable in the real-world applications.

Now, before you run to rewrite all your servlets to the asynchronous servlets – hold your horses for a minute. The solution works perfectly on a subset of use cases, such as the group chat notifications and auction house price alerts. You will most likely not benefit anything for the cases where the requests are waiting behind unique database queries being completed. So as always, I must reiterate my favourite performance-related recommendation – measure everything. Do not guess anything.

But on the occasions when the problem does fit the solution shape, I can only praise it. Besides the now-obvious improvements on throughput and latency, we have elegantly avoided possible thread starvation issues under heavy load.

Another important aspect – the approach to asynchronous request processing is finally standardized. Independent of your favourite Servlet API 3.0 – compliant application server such as Tomcat 7, JBoss 6 or Jetty 8 you can be sure the approach works. No more wrestling with different Comet implementations or platform-dependent solutions, such as the Weblogic FutureResponseServlet.
 

Reference: How to use Asynchronous Servlets to improve performance from our JCG partner Nikita Salnikov Tarnovski at the Plumbr Blog blog.
Related Whitepaper:

Functional Programming in Java: Harnessing the Power of Java 8 Lambda Expressions

Get ready to program in a whole new way!

Functional Programming in Java will help you quickly get on top of the new, essential Java 8 language features and the functional style that will change and improve your code. This short, targeted book will help you make the paradigm shift from the old imperative way to a less error-prone, more elegant, and concise coding style that’s also a breeze to parallelize. You’ll explore the syntax and semantics of lambda expressions, method and constructor references, and functional interfaces. You’ll design and write applications better using the new standards in Java 8 and the JDK.

Get it Now!  

4 Responses to "How to use Asynchronous Servlets to improve performance"

  1. patrick says:

    The problem with this code and your numbers is that the code for the two different implementation do two different things.

    In the synchronous example, you have a Thread.sleep that is executed on every single request.

    In the asynchronous example all your requests are executed by a single thread. The thread.sleep is only executed on the first request, before you enter the loop that drains the queue.

    This bug is what changes your performance numbers, it has nothing to do with your choice of I/O strategy

  2. Golthiryus says:

    @patrick Exaclty. There are programs that do different things. It has no sense to compare them.

  3. tom says:

    Hey Nikita, check out http://docs.oracle.com/javaee/7/api/index.html?javax/servlet/SingleThreadModel.html.
    You achieved similar thing with blockingqueue. All other threads waits to be executed.

  4. nav says:

    Note: Need to add the following annotation to the Work class to get this code working.
    @WebListener

Leave a Reply


8 − = two



Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.

Sign up for our Newsletter

20,709 insiders are already enjoying weekly updates and complimentary whitepapers! Join them now to gain exclusive access to the latest news in the Java world, as well as insights about Android, Scala, Groovy and other related technologies.

As an extra bonus, by joining you will get our brand new e-books, published by Java Code Geeks and their JCG partners for your reading pleasure! Enter your info and stay on top of things,

  • Fresh trends
  • Cases and examples
  • Research and insights
  • Two complimentary e-books