Tag Archives: Kafka

This is the third part of the article series: Processing real-time data with Storm, Kafka, and ElasticSearch. 1. Introduction In the second part, we learned how to perform searches in ElasticSearch. However, we failed to import the .json flight data files to ElasticSearch using its bulk API. In this article, we will do some programming, and learn some ways on ...

Previously we created successfully a Zookeeper ensemble, now it’s time to add some Kafka brokers that will connect to the ensemble and we shall execute some commands. We will pick up from the same docker compose file we compiled previously. First let’s jump on the configuration that a Kafka broker needs. 01 02 03 04 05 06 07 08 09 ...

Previously we spun up Zookeeper and Kafka locally but also through Docker. What comes next is spinning up more than just one Kafka and Zookeeper node and create a 3 node cluster. To achieve this the easy way locally docker-compose will be used. Instead of spinning up various instances on the cloud or running various Java processes and altering configs, ...

This is the second part of the article series: Processing real-time data with Storm, Kafka, and ElasticSearch. 1. Introduction In the first part we described the problem and how we are going to solve it. To refresh your memory, the plan is to create a Data Reduction System of historic flight data (which you can freely download from here). We ...

Kafka popularity increases every day more and more as it takes over the streaming world. It is already provided out of the box on cloud providers like AWS, Azure and IBM Cloud. Eventually for cases of local development it is a bit peculiar due to requiring various moving parts. This blog will focus on making it easy for a developer ...

This is an article of processing real-time data with Storm, Kafka and ElasticSearch. 1. Introduction How would you process a stream of real or near-real time data? In the era of Big Data, there are a number of technologies available that can help you in this task. In this series of articles we shall see a real example scenario and ...

Apache Avro, along with Apache Thrift and Protocol Buffers, is often being used as a platform-neutral extensible mechanism for serializing structured data. In the context of event-driven systems, the Apache Avro‘s schemas play the role of the language-agnostic contracts, shared between loosely-coupled components of the system, not necessarily written using the same programming language. Probably, the most widely adopted reference ...

Problem Moving from Kafka mirror maker to Brooklin has its advantages written by me here. But doing this migration is not easy-breezy as it should have been. Major challenge I faced was: Making SSL connection between consumer Kafka broker and Brooklin Solution SSL Problem This problem turned out to be more tricker than I had anticipated. I have been using ...

In distributed environment failure is a very common scenario that can be happened any time. In the Kafka environment, the broker can crash, network failure, failure in processing, failure while publishing message or failure to consume messages, etc. These different scenarios introduced a different kind of data loss and duplication. Failure scenarios A (Ack Failed): Producer published message successfully with retry>1 ...