Home » Java » Enterprise Java » Retention period and issue with Kafka data not getting deleted

About Abhijeet Iyengar

Abhijeet Iyengar
Abhijeet is a Software Engineer working with financial client . He has been involved in building UI and service based applications.

Retention period and issue with Kafka data not getting deleted

Problem

Default value of retention.ms attribute on kafka topis is set to 7 days but data still persist for pre 7 days in topic.

Version of kafka : 2.1.11

An interesting problem was that even after data being older then 7 days and retention.ms attributes not being overridden ( kept at 7 days ) still we were able to see data in topic older then that.

This is normally OK but for some scenarios where kafka topics are used as source of truth to create a in memory cache on startup of application, this could lead to problem:

  • As they have to read more data on startup
  • Might end up having more data in cache

The problem emerges due the fact of another parameter which is not spoken much segment.ms . This parameter plays a major role.

This parameter decides when does the internal segment of topic gets role. By default again its 7 days.

Now the log cleaner thread only deletes the segment when the last message in a given segment is older then 7 days. So if last message arrived on Saturday and lets say segment rolled on Sunday ( after the week ), the whole segment data ( from last weeks Monday to Sunday) would be available till the next Saturday.

Solution

To resolve this it would be sufficient to set segment.ms parameter to 24 hours, so that the segments get rolled every day and as and when 1 week passes the old data gets deleted.

Published on Java Code Geeks with permission by Abhijeet Iyengar, partner at our JCG program. See the original article here: Retention period and issue with Kafka data not getting deleted

Opinions expressed by Java Code Geeks contributors are their own.

(0 rating, 0 votes)
You need to be a registered member to rate this.
Start the discussion Views Tweet it!
Do you want to know how to develop your skillset to become a Java Rockstar?
Subscribe to our newsletter to start Rocking right now!
To get you started we give you our best selling eBooks for FREE!
1. JPA Mini Book
2. JVM Troubleshooting Guide
3. JUnit Tutorial for Unit Testing
4. Java Annotations Tutorial
5. Java Interview Questions
6. Spring Interview Questions
7. Android UI Design
and many more ....
I agree to the Terms and Privacy Policy
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Inline Feedbacks
View all comments