Moving from Kafka mirror maker to Brooklin has its advantages written by me here. But doing this migration is not easy-breezy as it should have been. Major challenge I faced was: Making SSL connection between consumer Kafka broker and Brooklin
This problem turned out to be more tricker than I had anticipated.
I have been using 1.0.2 version of Brooklin for my work. And the problem i faced was i was unable to create a SSL connection between kafkaMirroringConnector ( connector used to create a Kafka consumer to be listening to Kafka Broker ) and Kafka Broker and problem lies in the limitation of the code. After going through code, I realized the only option was to fix it and create a new build.
Technically the change belongs to just 2 file.
Build datastream-kafka-connector module and replace the jar int he brooklin lib directory.
Now when creating a new brooklin task you could use kafkassl in place of kafka in source string.
Note: Another method which i tried and didn’t work for was the use of consumerFactoryClassName param. This can help you create custom consumer as per the code but it caused me more issues.
Published on Java Code Geeks with permission by Abhijeet Iyengar, partner at our JCG program. See the original article here: Challenges of moving from Kafka Mirror Maker to Brooklin for SSL Kafka Brokers
Opinions expressed by Java Code Geeks contributors are their own.