Scala

Akka Cluster with Docker containers

This article will show you how to build docker images that contain a single akka cluster application. You will be able to run multiple seed nodes and multiple cluster nodes. The code can be found on Github and will be available as a Typesafe Activator.

If you don’t know docker or akka

Docker is the new shiny star in the devops world. It lets you easily deploy images toany OS running docker, while providing an isolated environment for the applications running inside the container image.

Akka is a framework to build concurrent, resilient, distributed and scalable software systems. The cluster feature lets you distribute your Actors across multiple machines to achieve load balancing, fail-over and the ability to scale up and out.

The big picture

This is what the running application will look like. No matter where your docker containers will run at the end of the day. The numbers at the top left describe the starting order of the containers.

akka-docker-bigpicture

First you have to start your seed nodes, which will “glue” the cluster together. After the first node is started all following seed-nodes have to know the ip address of the initial seed node in order to build up a single cluster. The approach describe in this article is very simple, but easily configurable so you can use it with other provision technologies like chef, puppet or zookeeper.

All following nodes that get started need at least one seed-node-ip in order to join the cluster.

The application configuration

We will deploy a small akka application which only logs cluster events. The entrypoint is fairly simple:

object Main extends App {
 
  val nodeConfig = NodeConfig parse args
 
  // If a config could be parsed - start the system
  nodeConfig map { c =>
    val system = ActorSystem(c.clusterName, c.config)
 
    // Register a monitor actor for demo purposes
    system.actorOf(Props[MonitorActor], "cluster-monitor")
 
    system.log info s"ActorSystem ${system.name} started successfully"
  }
 
}

The tricky part is the configuration. First, the akka.remote.netty.tcp.hostname configuration needs to be set to the docker ip address. The port configuration is unimportant as we have unique ip address thanks  to docker. You can read more about docker networking here. Second, the seed nodes should add themselves to the akka.cluster.seed-nodes list. And at last, everything should be configurable through system properties and environment variables. Thanks to the Typesafe Config Library this is achievable (even with some sweat and tears).

  1. Generate a small commandline parser with scopt and the following two parameters:
    –seed flag which determines if  this node starting should act as a seed node
    ([ip]:[port])… unbounded list of [ip]:[port] which represent the seed nodes
  2. Split the configuration in three files
    1. application.conf which contains the common configuration
    2. node.cluster.conf contains only  the node specific configuration
    3. node.seed.conf contains only the seed-node specific configuration
  3. A class NodeConfig which orchestrates all settings and cli parameters in the right order and builds a Typesafe Config object.

Take a closer look at the NodeConfig  class. The core part is this:

// seed nodes as generated string from cli
(ConfigFactory parseString seedNodesString)
  // the hostname
  .withValue("clustering.ip", ipValue)
  // node.cluster.conf or node.seed.conf
  .withFallback(ConfigFactory parseResources configPath) 
  // default ConfigFactory.load but unresolved
  .withFallback(config)
  // try to resolve all placeholders (clustering.ip and clustering.port)
  .resolve

The part to resolve the IP address is a bit hacky, but should work in default docker environments. First the eth0 interfaces is searched and then the first isSiteLocalAddress is being returned. IP adresses in the following ranges are local172.16.xxx.xxx, 172.31.xxx.xxx , 192.168.xxx.xxx, 10.xxx.xxx.xxx.

The main cluster configuration is done inside the clustering section of the application.conf:

clustering {
  # ip = "127.0.0.1" # will be set from the outside or automatically
  port = 2551
  cluster.name = "application"
}

The ip adress will be filled by the algorithm describe above if nothing else is set. You can easily override all settings with system properties. e.g if you want to run a seed node and a cluster node inside your IDE without docker start both like this:

# the seed node
-Dclustering.port=2551 -Dclustering.ip=127.0.0.1 --seed
# the cluster node
-Dclustering.port=2552 -Dclustering.ip=127.0.0.1 127.0.0.1:2551

For sbt this looks like this:

# the seed node
sbt runSeed
# the cluster node
sbt runNode

The build

Next we build our docker image. The sbt-native-packager plugin recently added experimental docker support, so we only need to  configure our build to be docker-ready. First add the plugin to your plugins.sbt.

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "0.7.4")

Now we add a few required settings to our build.sbt. You should use sbt 0.13.5 or higher.

// adds start script and jar mappings
packageArchetype.java_application
 
// the docker maintainer. You could scope this to "in Docker"
maintainer := "Nepomuk Seiler"
 
// Short package description
packageSummary := s"Akka ${version.value} Server"

And now we are set. Start sbt and run docker:publishLocal and a docker image will be created for you. The Dockerfile is in target/docker if you want to take a closer look what’s created.

Running the cluster

Now it’s time to run our containers. The image name is by default name:version. For the our activator it’s akka-docker:2.3.4. The seed ip adresses may vary. You can read it out of the console output of your seed nodes.

docker run -i -t -p 2551:2551 akka-docker:2.3.4 --seed
docker run -i -t -p 2551:2551 akka-docker:2.3.4 --seed 176.16.0.18:2551
docker run -i -t -p 2551:2551 akka-docker:2.3.4 176.16.0.18:2551 176.16.0.19:2551
docker run -i -t -p 2551:2551 akka-docker:2.3.4 176.16.0.18:2551 176.16.0.19:2551

What about linking?

This blog entry describes a different approach to build an akka cluster with docker. I used some of the ideas, but the basic concept is build ontop of linking the docker contains. This allows you to get the ip and port information of the running seed nodes. While this is approach is suitable for single host machines, it seems to get more messy when working with multiple docker machines.

The setup in this blog requires only one thing: A central way of assigning host ips. If your seed nodes don’t change their IP adresses you can basically configure almost everything already in your application.conf.

Reference: Akka Cluster with Docker containers from our JCG partner Nepomuk Seiler at the mukis.de blog.

Nepomuk Seiler

Nepomuk Seiler is a information sience student at the Ludwigs-Maximilians-University Munich. He uses Scala for machine learning tasks and web development with play. Also he provided some small contributions to the Eclipse community.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Back to top button