Apache Kafka provides reliable message consumption by allowing consumers to periodically commit their offsets. Offset commits inform Kafka that a consumer has successfully processed records up to a specific position. While offset commits appear straightforward, developers frequently encounter CommitFailedException when building long-running or high-throughput consumer applications. This exception is often confusing because it usually appears after the message processing has already completed successfully. In this article, we’ll understand what causes CommitFailedException, why Kafka throws it, and several practical techniques to prevent it. Finally, we’ll build a complete Java example that demonstrates both the incorrect and recommended approaches.
1. Understanding CommitFailedException in Kafka
CommitFailedException is thrown when a consumer attempts to commit offsets but is no longer the active owner of the partition. In other words, Kafka rejects the commit because another consumer has already taken ownership of the partition after a consumer group rebalance. A typical exception looks similar to the following:
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms.
Notice that Kafka clearly identifies the primary reason—the consumer failed to call poll() within the configured interval.
2. How Kafka Consumer Rebalancing Leads to CommitFailedException?
Kafka consumer groups are coordinated by the group coordinator. Every consumer must periodically call poll() to indicate that it is still alive and capable of processing records. If a consumer becomes unresponsive for too long, Kafka assumes that it has failed and initiates a rebalance. Once the rebalance completes:
- The partitions are assigned to another consumer.
- The original consumer loses ownership.
- Any subsequent offset commit from the old consumer is rejected.
- Kafka throws
CommitFailedException.
2.1 Common Causes of CommitFailedException
- Long-Running Message Processing: The most common cause of
CommitFailedExceptionis that message processing takes longer than the configuredmax.poll.interval.ms. After the consumer callspoll(), it may receive a large batch of records (for example, 500 records). If processing those records takes several minutes before the nextpoll()is invoked, Kafka assumes the consumer has become unresponsive and initiates a consumer group rebalance. By the time the application callscommitSync(), the partitions have already been reassigned to another consumer, causing Kafka to reject the offset commit and throw aCommitFailedException. - Processing Large Batches of Records: Configuring a very large
max.poll.recordsvalue may cause hundreds or even thousands of records to be returned in a single polling cycle. Processing the entire batch before callingpoll()again can easily exceed the configured poll interval, increasing the likelihood of a consumer rebalance and a subsequentCommitFailedException. - Slow Database or External Service Calls: Applications that perform time-consuming database updates, invoke slow REST APIs, execute machine learning inference, or process large files can significantly delay the next
poll()invocation. As a result, Kafka may consider the consumer unresponsive and remove it from the consumer group. - Latency Caused by Downstream Systems: When dependent systems such as databases, message brokers, or external services become slow or unavailable, message processing time increases considerably. If the consumer cannot return to
poll()within the configured interval, Kafka initiates a rebalance and any subsequent offset commit may fail. - Long Garbage Collection (GC) Pauses: Long Full GC pauses or severe memory pressure can temporarily suspend the JVM, preventing the consumer from polling Kafka. During these pauses, Kafka may assume the consumer has failed and reassign its partitions to another consumer, resulting in a
CommitFailedExceptionwhen offset commits are attempted.
2.2 Best Practices to Prevent CommitFailedException
- Keep Message Processing Fast: Try to complete processing quickly and return to
poll()as soon as possible. Avoid long blocking operations inside the consumer thread. - Reduce the
max.poll.recordsValue: Instead of processing 500 or 1000 records at once, reduce the batch size. Smaller batches complete much faster.props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 20);
- Increase the
max.poll.interval.msValue: If long processing is unavoidable, increase the poll interval.props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 900000);
Although this reduces the likelihood of the exception, it should not be treated as the primary solution because extremely large intervals delay failure detection.
- Offload Processing to Worker Threads: This is one of the most recommended production approaches. The consumer thread continues polling Kafka while worker threads process the data.
- Pause and Resume Partition Consumption: Kafka allows temporarily pausing partitions while processing is in progress. This prevents additional records from being fetched.
consumer.pause(consumer.assignment()); // processing consumer.resume(consumer.assignment());
- Commit Offsets More Frequently: Instead of committing only after processing a very large batch, commit offsets after smaller logical units of work.
3. Practical Example: Reproducing and Avoiding CommitFailedException
Before running the examples in this section, ensure that Apache Kafka is installed and running on your local machine. The consumer examples require an active Kafka broker to connect, along with a topic named orders for consuming messages. You can download Kafka from the official Apache Kafka website and start the Kafka broker locally. Once Kafka is running, create the required topic using the following command:
kafka-topics.sh --create \ --topic orders \ --bootstrap-server localhost:9092 \ --partitions 3 \ --replication-factor 1
After creating the topic, you can publish sample messages using a Kafka producer and run the Java consumer examples to observe normal processing as well as the scenarios that lead to CommitFailedException.
3.1 Adding the Kafka Client Dependency
Before creating a Kafka consumer application, add the Kafka Client library to your project. This dependency provides the core APIs required to create Kafka producers and consumers, manage consumer groups, perform offset commits, and interact with a Kafka cluster. Replace stable__jar__version with the latest stable version of the Kafka client that matches your Kafka broker version.
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>stable__jar__version</version>
</dependency>
3.2 Consumer Implementation That Triggers CommitFailedException
import org.apache.kafka.clients.consumer.*;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class CommitFailedExample {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "demo-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "10000");
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "100");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singleton("orders"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
for (ConsumerRecord<String, String> record : records) {
System.out.println("Processing : " + record.value());
Thread.sleep(15000);
}
consumer.commitSync();
}
}
}
In this example, we configure a Kafka consumer with manual offset commits by setting enable.auto.commit to false, allowing the application to explicitly control when offsets are committed. The consumer subscribes to the orders topic and continuously polls Kafka every second for new records. To intentionally reproduce CommitFailedException, the max.poll.interval.ms property is configured to just 10 seconds, while each record is artificially processed for 15 seconds using Thread.sleep(15000). Since the consumer does not invoke poll() again within the configured interval, Kafka assumes the consumer has become unresponsive and triggers a consumer group rebalance, assigning its partitions to another consumer. After processing completes, the application invokes commitSync(); however, because the consumer no longer owns the partitions, Kafka rejects the offset commit and throws a CommitFailedException. Although the code intentionally demonstrates the problem, it closely resembles real-world scenarios where long-running database operations, slow REST API calls, large batch processing, or CPU-intensive business logic delay polling long enough to cause a rebalance.
3.2.1 Code Explanation
The application first creates the consumer configuration, disables automatic offset commits, and sets a small max.poll.interval.ms value to make the timeout easier to reproduce. It then subscribes to the orders topic and enters an infinite polling loop. Each invocation of poll() retrieves a batch of records from Kafka, after which every record is processed with an intentional 15-second delay. Because the processing time exceeds the configured poll interval, the consumer fails to send another poll() request in time, causing Kafka to remove it from the consumer group during a rebalance. Finally, when commitSync() attempts to commit the processed offsets, Kafka rejects the request since the partitions have already been reassigned, resulting in a CommitFailedException.
3.2.2 Code Output
Processing : Order-101 Processing : Order-102 Exception in thread "main" org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms.
When the application starts, it successfully polls records from the orders topic and begins processing them one by one. Due to the intentional 15-second delay for each record, the consumer exceeds the configured max.poll.interval.ms of 10 seconds without invoking another poll(). Kafka interprets this as a failed or unresponsive consumer, initiates a consumer group rebalance, and transfers partition ownership to another consumer in the same group. After processing completes, the application attempts to commit the offsets using commitSync(), but the commit fails because the consumer is no longer the partition owner. As a result, the application prints the processed records followed by a CommitFailedException, clearly indicating that the consumer group has already rebalanced and the offset commit cannot be completed.
3.3 Consumer Implementation That Avoids CommitFailedException
The following example demonstrates a safer configuration using smaller batches, a longer poll interval, and faster processing.
import org.apache.kafka.clients.consumer.*;
import java.time.Duration;
import java.util.Collections;
import java.util.Properties;
public class SafeConsumerExample {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "safe-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "10");
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, "300000");
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Collections.singleton("orders"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
for (ConsumerRecord<String, String> record : records) {
System.out.println("Processing : " + record.value());
process(record);
}
consumer.commitSync();
}
}
private static void process(ConsumerRecord<String, String> record) {
System.out.println("Completed : " + record.offset());
}
}
Unlike the previous example, this implementation follows several best practices to minimize the chances of encountering a CommitFailedException. The consumer is configured to process a smaller number of records by setting max.poll.records to 10, ensuring that each polling cycle completes quickly. Additionally, max.poll.interval.ms is increased to 300000 (5 minutes), providing sufficient time for legitimate business processing before Kafka considers the consumer unresponsive. The application still uses manual offset commits by disabling automatic commits, but it processes each record immediately through the process() method and commits the offsets only after the entire batch has been successfully processed. By reducing batch size, avoiding long-running operations in the polling loop, and allowing an appropriate poll interval, the consumer remains an active member of the consumer group and can safely commit offsets without triggering unnecessary rebalances.
3.3.1 Code Explanation
The application creates a Kafka consumer with manual offset management and subscribes to the orders topic. During each iteration, the consumer invokes poll() to fetch up to ten records, limiting the amount of work performed in a single polling cycle. Each record is then processed immediately by the process() method, which represents the application’s business logic. Because the processing is lightweight and the consumer returns to poll() quickly, Kafka continues to receive regular heartbeat and polling activity, preventing the consumer from being removed from the consumer group. Once all records in the current batch have been processed successfully, the application invokes commitSync() to persist the offsets, ensuring that messages are not reprocessed if the consumer restarts. This approach significantly reduces the likelihood of consumer group rebalances caused by delayed polling and enables reliable offset management.
3.3.2 code Output
Processing : Order-101 Completed : 0 Processing : Order-102 Completed : 1 Processing : Order-103 Completed : 2 Offsets committed successfully.
When the application executes, the consumer continuously polls small batches of records from the orders topic, processes each message successfully, and prints both the message value and its corresponding offset. Since the processing completes well within the configured max.poll.interval.ms, the consumer remains an active member of the consumer group throughout its execution. After processing each batch, commitSync() commits the latest offsets without any errors, allowing the application to continue processing new records normally. Unlike the previous example, no consumer rebalance occurs due to delayed polling, and therefore no CommitFailedException is thrown.
4. Conclusion
CommitFailedException is not caused by offset commits themselves but by the loss of partition ownership following a consumer group rebalance. The most common trigger is failing to invoke poll() within the configured max.poll.interval.ms, often due to long-running business logic, large batch sizes, or slow downstream dependencies. Simply increasing the poll interval may reduce the frequency of the exception, but the most effective solution is to design consumers that remain responsive by keeping the polling thread lightweight, limiting batch sizes, and offloading expensive processing to worker threads. By understanding Kafka’s consumer group lifecycle and following these best practices, applications can process messages reliably while avoiding unnecessary rebalances and offset commit failures in production environments.
