Enterprise Java

Apache Gearpump

Apache Gearpump is a real-time big data streaming engine. It conceived at Intel in mid-2014 as an open source project on GitHub from start and entered Apache incubation on March 8th, 2016. The name Gearpump is a reference to the engineering term “gear pump” which is a super simple pump that consists of only two gears, but is very powerful at streaming water. Different to other streaming engines, Gearpump’s engine is event/message based. Per initial benchmarks we are able to process 18 million messages per second (message length is 100 bytes) with a 8ms latency on a 4-node cluster.

The Highlights

  • Extremely high throughput and low latency stream processing
  • Configurable message delivery guarantee (at least once, exactly once)
  • Application hot re-deployment
  • Comprehensive Dashboard for application monitoring
  • Native Storm Application compatibility
  • Native Samoa Application compatibility
  • Friendly and extensible APIs

Before you can submit and run your first Gearpump application, you will need a running Gearpump service. There are multiple ways to run Gearpump Local mode, Standalone mode, YARN mode or Docker mode.

The easiest way is to run Gearpump in Local mode. Any Linux, MacOSX or Windows desktop can be used with zero configuration.

In the example below, we assume your are running in Local mode. If you running Gearpump in one of the other modes, you will need to configure the Gearpump client to connect to the Gearpump service by setting the gear.conf configuration path in classpath. Within this file, you will need to change the parameter gearpump.cluster.masters to the correct Gearpump master(s).

Steps to submit your first Application

Step 1: Submit Application

After the cluster is started, you can submit an example wordcount application to the cluster

Open another shell,

### To run WordCount example
bin/gear app -jar examples/wordcount-2.11-0.8.4-assembly.jar org.apache.gearpump.streaming.examples.wordcount.WordCount

Step 2: View Application Status and Metrics.

To view the application status and metrics, start the Web UI services, and browse to to check the status. The default username and password is “admin:admin”, you can check UI Authentication to find how to manage users.

[1] https://gearpump.apache.org/overview.html

[2] https://www.slideshare.net/manuzhang/apache-gearpump-lightweight-runtime-streaming-engine

[3] http://incubator.apache.org/projects/gearpump.html

[4] https://gearpump.apache.org/releases/latest/introduction/submit-your-1st-application/index.html

Published on Java Code Geeks with permission by Furkan Kamaci, partner at our JCG program. See the original article here: Apache Gearpump

Opinions expressed by Java Code Geeks contributors are their own.

Furkan Kamaci

Furkan KAMACI is a Machine Learning, NLP and Search Expert who loves Java! He works at Alcatel - Lucent as Integration Professional.
Notify of

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Inline Feedbacks
View all comments
Back to top button