Tag Archives: Kafka

1. Introduction This series of articles shows how one can process real time stream data using a number of technologies. The input data stream is flight data arriving real-time from a sensor (either one you can buy from Amazon or more advanced ones like civilian or military radars). To avoid the hassle of buying a sensor or connecting to one ...

This is the third part of the article series: Processing real-time data with Storm, Kafka, and ElasticSearch. 1. Introduction In the second part, we learned how to perform searches in ElasticSearch. However, we failed to import the .json flight data files to ElasticSearch using its bulk API. In this article, we will do some programming, and learn some ways on ...

Previously we created successfully a Zookeeper ensemble, now it’s time to add some Kafka brokers that will connect to the ensemble and we shall execute some commands. We will pick up from the same docker compose file we compiled previously. First let’s jump on the configuration that a Kafka broker needs. 01 02 03 04 05 06 07 08 09 ...

Previously we spun up Zookeeper and Kafka locally but also through Docker. What comes next is spinning up more than just one Kafka and Zookeeper node and create a 3 node cluster. To achieve this the easy way locally docker-compose will be used. Instead of spinning up various instances on the cloud or running various Java processes and altering configs, ...

This is the second part of the article series: Processing real-time data with Storm, Kafka, and ElasticSearch. 1. Introduction In the first part we described the problem and how we are going to solve it. To refresh your memory, the plan is to create a Data Reduction System of historic flight data (which you can freely download from here). We ...

Kafka popularity increases every day more and more as it takes over the streaming world. It is already provided out of the box on cloud providers like AWS, Azure and IBM Cloud. Eventually for cases of local development it is a bit peculiar due to requiring various moving parts. This blog will focus on making it easy for a developer ...

This is an article of processing real-time data with Storm, Kafka and ElasticSearch. 1. Introduction How would you process a stream of real or near-real time data? In the era of Big Data, there are a number of technologies available that can help you in this task. In this series of articles we shall see a real example scenario and ...

Apache Avro, along with Apache Thrift and Protocol Buffers, is often being used as a platform-neutral extensible mechanism for serializing structured data. In the context of event-driven systems, the Apache Avro‘s schemas play the role of the language-agnostic contracts, shared between loosely-coupled components of the system, not necessarily written using the same programming language. Probably, the most widely adopted reference ...

Problem Moving from Kafka mirror maker to Brooklin has its advantages written by me here. But doing this migration is not easy-breezy as it should have been. Major challenge I faced was: Making SSL connection between consumer Kafka broker and Brooklin Solution SSL Problem This problem turned out to be more tricker than I had anticipated. I have been using ...