Tag Archives: Kafka

Apache Avro, along with Apache Thrift and Protocol Buffers, is often being used as a platform-neutral extensible mechanism for serializing structured data. In the context of event-driven systems, the Apache Avro‘s schemas play the role of the language-agnostic contracts, shared between loosely-coupled components of the system, not necessarily written using the same programming language. Probably, the most widely adopted reference ...

Problem Moving from Kafka mirror maker to Brooklin has its advantages written by me here. But doing this migration is not easy-breezy as it should have been. Major challenge I faced was: Making SSL connection between consumer Kafka broker and Brooklin Solution SSL Problem This problem turned out to be more tricker than I had anticipated. I have been using ...

In distributed environment failure is a very common scenario that can be happened any time. In the Kafka environment, the broker can crash, network failure, failure in processing, failure while publishing message or failure to consume messages, etc. These different scenarios introduced a different kind of data loss and duplication. Failure scenarios A (Ack Failed): Producer published message successfully with retry>1 ...

Extract transform load is process for pulling data from one datasystem and loading into another datasystem. Datasystem involved are called source system and target system. Shape of data from source system does not match to the target system, so some conversion is required to make it compatible and that process is called transformation. Transformation is made of map/filter/reduce operations. To ...

For a project, I’m trying to log the basic transactions of the user such as addition and removal of an item and for multiple types of items and sending a message to kafka for each transaction. The accuracy of the log mechanism is not crucial and I don’t want it to block my business code in the case of kafka ...

Publish-Subscribe Messaging systems play an important role in any enterprise architecture as it enables reliable integration without tightly coupling the applications. The ability to share data between decoupled systems is not a problem that is easily tackled. Consider an enterprise with multiple applications that are being built independently, with different languages and platforms. It needs to share data and processes ...

Why would a data scientist use Kafka Jupyter Python KSQL TensorFlow all together in a single notebook? There is an impedance mismatch between model development using Python and its Machine Learning tool stack and a scalable, reliable data platform. The former is what you need for quick and easy prototyping to build analytic models. The latter is what you need ...

Overview This sample project demonstrates how to build real-time streaming applications using event-driven architecture, Spring Boot,Spring Cloud Stream, Apache Kafka and Lombok. By the end of this tutorial you’ll have a simple Spring Boot based Greetings micro-service running that takes a message from a REST api writes it to a Kafka topic reads it from the topic outputs it to the console Let’s get started! Btw, you’ll ...

Introduction Apache Flink is an open source platform for distributed stream and batch data processing. Flink is a streaming data flow engine with several APIs to create data streams oriented application. It is very common for Flink applications to use Apache Kafka for data input and output. This article will guide you into the steps to use Apache Flink with ...