Home » Software Development » The simple scalability equation

Vlad Mihalcea is a software architect passionate about software integration, high scalability and concurrency challenges.

The simple scalability equation

Queueing Theory

The queueing theory allows us to predict queue lengths and waiting times, which is of paramount importance for capacity planning. For an architect this is a very handy tool, since queues are not just the appanage of messaging systems.

To avoid system over loading we use throttling. Whenever the number of incoming requests surpasses the available resources, we basically have two options:

• discarding all overflowing traffic, therefore decreasing availability
• queuing requests and wait (for as long as a time out threshold) for busy resources to become available

This behaviour applies to thread-per-request web servers, batch processors or connection pools.

What’s in it for us?

Agner Krarup Erlang is the father of queueing theory and traffic engineering, being the first to postulated the mathematical models required to provisioning telecommunication networks.

Erlang formulas are modelled for M/M/k queue models, meaning the system is characterized by:

The Erlang formulas give us the servicing probability for:

This is not strictly applicable to thread pools, as requests are not fairly serviced and servicing times not always follow an exponential distribution.

A general purpose formula, applicable to any stable system (a system where the arrival rate is not greater than the departure rate) is Little’s Law.

$latex.php$

where

L – average number of customers
λ – long-term average arrival rate
W – average-time a request spends in a system

You can apply it almost everywhere, from shoppers queues to web request traffic analysis.

This can be regarded as a simple scalability formula, for to double the incoming traffic we have two options:

1. reduce by half the response time (therefore increasing performance)
2. double the available servers (therefore adding more capacity)

A real life example

A simple example is a super-market waiting line. When you arrive at the line up you must pay attention to the arrival rate (e.g. λ = 2 persons / minute) and the queue length (e.g. L = 6 persons) to find out the amount of time you are going to spend waiting to be served (e.g. W = L / λ = 3 minutes).

A provisioning example

Let’s say we want to configure a connection pool to support a given traffic demand.
The connection pool system is characterized by the following variables:

Ws = service time (the connection acquire and hold time) = 100 ms = 0.1s
Ls = in-service requests (pool size) = 5

Assuming there is no queueing (Wq = 0):

Our connection pool can deliver up to 50 requests per second without ever queueing any incoming connection request.

Whenever there are traffic spikes we need to rely on a queue, and since we impose a fixed connection acquire timeout the queue length will be limited.

Since the system is considered stable the arrival rate applies both to the queue entry as for the actual services:

This queuing configuration still delivers 50 requests per second but it may queue 100 requests for 2 seconds as well, so a traffic burst of 150 requests lasting for 1 second would be manageable, since 50 requests can be served in the first second and the other 100 in following 2 seconds.

 Reference: The simple scalability equation from our JCG partner Vlad Mihalcea at the Vlad Mihalcea’s Blog blog.
Do you want to know how to develop your skillset to become a Java Rockstar?
Subscribe to our newsletter to start Rocking right now!
To get you started we give you our best selling eBooks for FREE!
1. JPA Mini Book
2. JVM Troubleshooting Guide
3. JUnit Tutorial for Unit Testing
4. Java Annotations Tutorial
5. Java Interview Questions
6. Spring Interview Questions
7. Android UI Design
and many more ....

Be the First to Comment!

Subscribe
Notify of
Want to take your Java skills to the next level?
Here are some of the eBooks you will get:
• Spring Interview QnA
• JPA Minibook
• JVM Troubleshooting Guide
• Java Interview QnA
• Java Design Patterns
The Spring Framework Cookbook
• Learn the best practices of the Spring Framework
• Build simple, portable, fast and flexible JVM-based systems and applications
• Explore specific projects like Boot and Batch
The Spring Data Programming Cookbook
• Learn how to use data access technologies and cloud-based data services
• Set up the environment and create a basic project
• Learn how to handle the various modules (e.g. JPA, MongoDB, Redis etc.)
The Selenium Programming Cookbook
• Kick-start your own projects using this testing framework for web applications
• Learn JUnit integration and Standalone Server functionality
• Find out the most popular Interview Questions about the Selenium Framework
The Mockito Programming Cookbook
• Kick-start your own web projects using this open source testing framework
• Write simple test cases using the Mockito Framework
• Learn how to integrate with JUnit, Maven and other frameworks
The JUnit Programming Cookbook
• Learn basic usage and configuration of JUnit
• Learn how to integrate with other testing frameworks
The JSF 2.0 Programming Cookbook
• Build component-based user interfaces for web applications
• Set up the environment and create a basic project
• Learn Internationalization and Facelets Templates
The Amazon S3 Tutorial
• Develop your own Amazon S3 based applications
• Learn API usage and pricing
• Get your own projects up and running in minimum time
Java Design Patterns
• Learn how Design Patterns are implemented and utilized in Java
• Understand the reasons why patterns are so important
• Learn when and how to apply each one of them
Java Concurrency Essentials
• Dive into the magic of concurrency
• Learn about testing concurrent applications
The IntelliJ IDEA Handbook
• Kick-start your own programming projects using IntelliJ IDEA
• Learn how to setup and install plugins
• Create UIs with this Java integrated development environment
The Git Tutorial
• Learn why Git differs from other version control systems
• Explore Git's usage and best practises
• Learn branching strategies
The Eclipse IDE Handbook
• Explore the most widely used Java IDE
• Learn how to setup and install plugins
• Built your own projects up and running in minimum time
The Docker Containerization Cookbook
• Explore the world’s leading software containerization platform
• Learn how to wrap a piece of software in a complete filesystem
• Learn how to use DNS and various commands
Developing Modern Applications With Scala
• Develop modern Scala applications
• Build SBT and reactive applications
• Learn about testing and database access
The Apache Tomcat Cookbook
• Explore Apache Tomcat open-source web server
• Learn about installation, configuration, logging and clustering
• Kick-start your own web projects using Apache Tomcat
The Apache Maven Cookbook
• Explore the Apache Maven build automation tool
• Learn about Maven's project structure and configuration
• Learn about Maven's dependency management and plug-ins
• Explore the Apache Hadoop open-source software framework
• Learn distributed caching and streaming
The Android Programming Cookbook
• Explore the Android mobile operating system
• Learn about services and page views
The Elasticsearch Tutorial
• Explore the Elasticsearch search engine
• Develop your own Elasticsearch based applications
• Learn operations, Java API Integration and reporting
Amazon DynamoDB Tutorial
• Develop your own Amazon DynamoDB based applications
• Learn DynamoDB Concepts and Best Practices
• Get your own projects up and running in minimum time
Java NIO Programming Cookbook
• Learn features for intensive I/O operations
• Follow a series of tutorials on Java NIO examples
• Get knowledge on Java Nio Socket and Asynchronous Channels
JBoss Drools Cookbook
• Explore Drools business rule management system
• Follow a series of tutorials on Drools examples
• Get knowledge on business rules for a shopping domain model
• Explore Vaadin web framework for rich Internet applications
• Learn the Architecture and Best Practices
• Get knowledge on Data Binding and Custom Components
Groovy Programming Cookbook
• Explore Apache Groovy object-oriented programming language
• Create sample applications and explore interview questions
• Get knowledge on Callback functionality and various widgets
GWT Programming Cookbook
• Explore the open source Google Web Toolkit
• Create sample applications and explore interview questions
• Create and maintain complex JavaScript front-end applications in Java
Do you want to know how to develop your skillset to become a Java Rockstar?
Subscribe to our newsletter to start Rocking right now!
To get you started we give you our best selling eBooks for FREE!
1. JPA Mini Book
2. JVM Troubleshooting Guide
3. JUnit Tutorial for Unit Testing
4. Java Annotations Tutorial
5. Java Interview Questions
and many more ....
Do you want to know how to develop your skillset to become a Java Rockstar?
Subscribe to our newsletter to start Rocking right now!
To get you started we give you our best selling eBooks for FREE!
1. JPA Mini Book
2. JVM Troubleshooting Guide
3. JUnit Tutorial for Unit Testing
4. Java Annotations Tutorial
5. Java Interview Questions
and many more ....
Do you want to know how to develop your skillset to become a Java Rockstar?
Subscribe to our newsletter to start Rocking right now!
To get you started we give you our best selling eBooks for FREE!
1. JPA Mini Book
2. JVM Troubleshooting Guide
3. JUnit Tutorial for Unit Testing
4. Java Annotations Tutorial
5. Java Interview Questions
6. Spring Interview Questions
7. Android UI Design
and many more ....
Want to be a DynamoDB Master ?
In order to help you master this Amazon NoSQL database service, we have compiled a kick-ass guide with all the major DynamoDB features and use cases! Besides studying them online you may download the eBook in PDF format!
Programming Interview Coming Up?
In order to get you prepared for your next Programming Interview, we have compiled a huge list of relevant Questions and their respective Answers. Besides studying them online you may download the eBook in PDF format!
Java Interview Coming Up?
In order to get you prepared for your next Java Interview, we have compiled a huge list of relevant Questions and their respective Answers. Besides studying them online you may download the eBook in PDF format!
Java Interview Coming Up?
In order to get you prepared for your next Java Interview, we have compiled a huge list of relevant Questions and their respective Answers. Besides studying them online you may download the eBook in PDF format!
Want to be a Java NIO Master ?
In order to help you master Java NIO Library, we have compiled a kick-ass guide with all the major Java NIO features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to be a Drools Master ?
In order to help you master Drools Business Rule Management System, we have compiled a kick-ass guide with all the major Drools features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to be an iText Master ?
In order to help you master iText Library, we have compiled a kick-ass guide with all the major iText features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to be an Elasticsearch Master ?
In order to help you master Elasticsearch search engine, we have compiled a kick-ass guide with all the major Elasticsearch features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to be a Scala Master ?
In order to help you master Scala, we have compiled a kick-ass guide with all the basic concepts! Besides studying them online you may download the eBook in PDF format!
Want to be a JUnit Master ?
In order to help you master unit testing with JUnit, we have compiled a kick-ass guide with all the major JUnit features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to master Amazon Web Services ?
In order to help you master the leading Web Services platform, we have compiled a kick-ass guide with all its major features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to master Spring Framework ?
In order to help you master the leading and innovative Java framework, we have compiled a kick-ass guide with all its major features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to master Eclipse IDE ?
In order to help you master Eclipse, we have compiled a kick-ass guide with all the basic features of the popular IDE! Besides studying them online you may download the eBook in PDF format!
Want to master IntelliJ IDEA ?
In order to help you master IntelliJ IDEA, we have compiled a kick-ass guide with all the basic features of the popular IDE! Besides studying them online you may download the eBook in PDF format!
Want to master Docker ?
In order to help you master Docker, we have compiled a kick-ass guide with all the basic concepts of the Docker container system! Besides studying them online you may download the eBook in PDF format!
Want to create a kick-ass Android App ?
With this book, you will delve into the fundamentals of Android programming. You will understand user input, views and layouts. Furthermore, you will learn how to communicate over Bluetooth and also leverage Google Maps into your application!
Want to be a GIT Master ?
In order to help you master GIT, we have compiled a kick-ass guide with all the basic concepts of the GIT version control system! Besides studying them online you may download the eBook in PDF format!
Want to be a Hadoop Master ?
In order to help you master Apache Hadoop, we have compiled a kick-ass guide with all the basic concepts of a Hadoop cluster! Besides studying them online you may download the eBook in PDF format!
Want to master Spring Data ?
In order to help you master Spring Data, we have compiled a kick-ass guide with all the major features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to create a kick-ass Android App ?
With this book, you will delve into the fundamentals of Android UI design. You will understand user input, views and layouts, as well as adapters and fragments. Furthermore, you will learn how to add multimedia to an app and also leverage themes and styles!
Want to be a Java 8 Ninja ?
In order to get you up to speed with the major Java 8 release, we have compiled a kick-ass guide with all the new features and goodies! Besides studying them online you may download the eBook in PDF format!
Want to master Java Annotations ?
In order to help you master the topic of Annotations, we have compiled a kick-ass guide with all the major features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to be a JUnit Master ?
In order to help you master unit testing with JUnit, we have compiled a kick-ass guide with all the major JUnit features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to master Java Abstraction ?
In order to help you master the topic of Abstraction, we have compiled a kick-ass guide with all the major features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to master Java Reflection ?
In order to help you master the topic of Reflection, we have compiled a kick-ass guide with all the major features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to be a JMeter Master ?
In order to help you master load testing with JMeter, we have compiled a kick-ass guide with all the major JMeter features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to be a Servlets Master ?
In order to help you master programming with Java Servlets, we have compiled a kick-ass guide with all the major servlet API uses and showcases! Besides studying them online you may download the eBook in PDF format!
Want to be a JAXB Master ?
In order to help you master XML Binding with JAXB, we have compiled a kick-ass guide with all the major JAXB features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to be a JDBC Master ?
In order to help you master database programming with JDBC, we have compiled a kick-ass guide with all the major JDBC features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to be a JPA Master ?
In order to help you master programming with JPA, we have compiled a kick-ass guide with all the major JPA features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to be a Hibernate Master ?
In order to help you master JPA and database programming with Hibernate, we have compiled a kick-ass guide with all the major Hibernate features and use cases! Besides studying them online you may download the eBook in PDF format!
Want to be a JSF Master ?
Want to be a Java Master ?
In order to help you master the Java programming language, we have compiled a kick-ass guide with all the must-know advanced Java features! Besides studying them online you may download the eBook in PDF format!
Want to be a Java Master ?
In order to help you master the Java programming language, we have compiled a kick-ass guide with all the must-know Design Patterns for Java! Besides studying them online you may download the eBook in PDF format!
Want to be a Hadoop Master ?
In order to help you master Apache Hadoop, we have compiled a kick-ass guide with all the basic concepts of a Hadoop cluster! Besides studying them online you may download the eBook in PDF format!
Want to be an Elastic Beanstalk Master ?
In order to help you master AWS Elastic Beanstalk, we have compiled a kick-ass guide with all the major Elastic Beanstalk features and use cases! Besides studying them online you may download the eBook in PDF format!
Amazon Elastic Beanstalk Tutorial
• Develop your own Amazon Elastic Beanstalk based applications
• Learn Java Integration and Command Line Interfacing
• Get your own projects up and running in minimum time
Want to take your Java skills to the next level?
Here are some of the eBooks you will get:
• Spring Interview QnA
• JPA Minibook
• JVM Troubleshooting Guide