After 14 years and 100,000+ lines of code, Apache Kafka finally cuts the cord. Here’s everything that changed, and what it means for your stack.
If you’ve run Kafka in production for any length of time, you know the ritual: stand up a ZooKeeper ensemble first, keep it in sync, monitor it separately, debug it when things go sideways. For over a decade, ZooKeeper was the silent dependency behind every Kafka cluster — managing metadata, coordinating leader elections, keeping brokers honest.
That era is over. Apache Kafka 4.0, released on March 18, 2025, ships with ZooKeeper mode fully removed. KRaft — Kafka’s own Raft-based metadata system — is now the only way to run a Kafka cluster. This is not a soft deprecation. ZooKeeper support is gone. And honestly? It’s been a long time coming.
1. What Is KRaft and Why Does It Matter?
KRaft stands for Kafka Raft — a consensus protocol built directly into Kafka, inspired by the Raft algorithm. The core idea is that Kafka no longer needs an external system to store and coordinate metadata. Instead, a dedicated set of controller nodes within the Kafka cluster itself handles all of that work using a replicated log.
Before KRaft, when a broker came online or a partition needed a new leader, ZooKeeper was the referee. Every metadata change, every controller election, every configuration update — it all flowed through ZooKeeper. That meant two systems to deploy, two systems to monitor, two systems to fail. With KRaft, metadata lives in Kafka itself, as a special internal topic. One system. Full stop.
“We no longer need to manage TLS between ZooKeeper and Kafka. We’re not managing compute for ZooKeeper anymore.” — Sandon Jacobs, Senior Developer Advocate, Confluent
1.1 KRaft vs ZooKeeper: Side-by-Side
| Dimension | ZooKeeper Mode (≤3.x) | KRaft Mode (4.0+) |
|---|---|---|
| Metadata storage | Separate ZooKeeper ensemble | Internal Kafka log on controller nodes |
| Deployment complexity | Two systems (Kafka + ZooKeeper) | One system |
| Max partitions (practical) | ~200,000 per cluster | ~1.9M per cluster (extrapolated) |
| Controller failover | ZooKeeper-coordinated, slower | Internal Raft election, faster |
| Dynamic controller changes | Not supported without restart | Add/remove controllers with no downtime |
| TLS scope | Kafka + separate ZK TLS config | Single Kafka TLS config |
| Kafka 4.0 support | Removed | Default & only mode |
Partition Capacity: KRaft vs ZooKeeper
2. The New Consumer Rebalance Protocol (KIP-848)
The second biggest change in Kafka 4.0 has nothing to do with ZooKeeper. KIP-848 introduces a completely redesigned consumer group rebalance protocol, and it’s now generally available.
The old protocol had a fundamental problem: when any consumer in a group joined, left, or crashed, the entire group would stop processing and go through a full rebalance — a “stop-the-world” event. Every member revoked every partition, the group leader calculated a new assignment, and then partitions were redistributed. In large deployments with frequent rolling restarts or auto-scaling, this caused serious latency spikes and throughput drops.
KIP-848 fixes this at the architectural level. The broker now manages partition assignment rather than delegating it to the consumer group leader. When a new consumer joins, it receives partitions incrementally — it doesn’t need to wait for everyone else to revoke and rejoin. Rebalances become cooperative and gradual rather than abrupt.
How to opt in
The new protocol is enabled automatically on the server side in Kafka 4.0. On the client side, consumers opt in by setting
group.protocol=consumerin their configuration. The legacy protocol (group.protocol=classic) remains available for backward compatibility.
3. Queues for Kafka — Early Access (KIP-932)
Kafka has always been a topic-based, pub/sub platform. That model is powerful, but it creates a hard constraint: you can’t have more active consumers in a group than partitions in the topic. Need 50 workers consuming a topic with 10 partitions? That’s not possible in classic Kafka — 40 of those consumers would sit idle.
KIP-932 introduces the concept of Share Groups — a new consumer group type that enables cooperative consumption across any number of consumers, regardless of partition count. Share groups support individual message acknowledgments, redelivery for failed messages, and unordered processing. This opens Kafka to use cases previously handled by traditional queue systems like RabbitMQ or Google Cloud Pub/Sub.
| Feature | Consumer Groups (classic) | Share Groups (KIP-932) |
|---|---|---|
| Max consumers | = number of partitions | Unlimited |
| Message ordering | Per-partition ordering | Unordered (within a partition) |
| Per-message ack | Not supported | Supported |
| Failed message retry | Requires custom logic | Built-in redelivery |
| Status in 4.0 | GA | Early Access |
KIP-932 is early access — not production-ready yet — but it signals a clear direction: Kafka is broadening its scope from an event streaming backbone to a full messaging platform.
4. What Else Changed in 4.0
4.1 Java Requirements Bumped
Kafka 4.0 raises the minimum Java bar across the board. Clients and Kafka Streams now require Java 11, and Brokers, Connect, and tooling require Java 17. If you’re still on Java 8, plan your upgrade path before touching Kafka 4.0.
4.2 Eligible Leader Replicas — KIP-966
KRaft now tracks Eligible Leader Replicas (ELR) — a subset of replicas that have complete data up to the high watermark but are not currently in the ISR. ELRs can be elected as leader safely, reducing the risk of data loss in failure scenarios. This ships as preview in 4.0.
4.3 Pre-Vote for Cleaner Leader Elections — KIP-996
In distributed systems, spurious leader elections are a source of unnecessary churn. KIP-996 introduces a pre-vote mechanism to KRaft: before triggering a full election, a candidate checks whether it can win. This reduces unnecessary elections and makes clusters more stable under network instability.
4.4 Deprecated APIs Removed
Kafka 4.0 follows its policy of removing APIs deprecated for at least 12 months. This includes support for message formats v0 and v1, and requires broker versions 2.1+ for Kafka 4.0 clients. Check the official upgrade docs for the full list before upgrading.
Kafka 4.0 Feature Readiness Overview
5. How to Migrate to Kafka 4.0
The migration path depends on where you’re starting from. The most important rule: you cannot upgrade directly from ZooKeeper mode to Kafka 4.0. You must first migrate to KRaft mode in a 3.x release, then upgrade to 4.0.
1.Upgrade to Kafka 3.9 first (the “bridge release”)
Kafka 3.9 is the recommended bridge to 4.0. It supports dynamic KRaft quorums (add/remove controllers without downtime) and has the ZooKeeper-to-KRaft migration tooling production-ready since 3.7. If you’re below 3.3, upgrade incrementally.
2. Migrate your cluster from ZooKeeper to KRaft mode
Use the official migration tooling introduced in 3.6. This is an in-place migration — no need for a full cluster rebuild. Follow the migration guide closely and test in staging first. Check for the zookeeper.metadata.migration.enable flag.
3. Finalize the metadata version
After confirming cluster health in KRaft mode, finalize the metadata version using the Kafka features tool. This is required before upgrading to 4.0.
4. Upgrade brokers to 4.0 one at a time
Perform a rolling upgrade — shut down one broker, update the binary, restart it, verify health. Then proceed to the next. Once all brokers are on 4.0, finalize the upgrade with the features script.
5. Run the features finalization command
Once all brokers are running 4.0, lock in the release version:
# Finalize the Kafka 4.0 upgrade after all brokers are updated bin/kafka-features.sh \ --bootstrap-server localhost:9092 \ upgrade \ --release-version 4.0
Important Downgrade Note
Once your consumer groups use the new KIP-848 protocol, you can only downgrade to Kafka 3.4.1 or newer — not to any older 3.x release. Plan accordingly before enabling it in production.
5.1 Opt into the New Consumer Protocol
After upgrading, enable the new consumer group protocol by adding a single config on the consumer side:
# In your consumer's application.properties or Kafka client config group.protocol=consumer # Verify your broker + client versions are compatible: # - Broker: Kafka 4.0+ # - Java clients: kafka-clients 4.0+
6. What This Means for Ops Teams
Beyond the technical changes, Kafka 4.0 meaningfully changes the operational reality for teams running Kafka in production. Removing ZooKeeper collapses what was a two-system monitoring problem into one. Fewer moving parts mean simpler runbooks, simpler alerting, and simpler on-call rotations.
Teams early to adopt reported tangible results: one fintech engineering team cut cluster setup time by roughly 40% after eliminating ZooKeeper from their workflow, while a SaaS provider reduced infrastructure costs by around 20% by retiring the dedicated ZooKeeper nodes and redistributing that capacity to Kafka brokers.
For teams on managed Kafka services (Confluent Cloud, AWS MSK, Google Managed Kafka), the transition will largely be handled transparently. If you’re self-managing, the migration work is real but well-documented, and the long-term operational gain is substantial.
7. What We’ve Learned
Kafka 4.0 is not a subtle release. It’s the culmination of 3.5+ years of work, 1,500 commits, and a deliberate effort to simplify one of the most widely deployed distributed systems in the world. The removal of ZooKeeper is the headline, but the full picture is richer than that.
We covered the KRaft architecture and why a single-system metadata model outperforms the two-system ZooKeeper approach — both in partition scale (up to ~1.9M partitions) and in operational simplicity. We looked at KIP-848’s cooperative consumer rebalance protocol, which eliminates stop-the-world rebalances and makes large consumer groups dramatically more stable. We explored KIP-932’s Share Groups, which bring queue semantics to Kafka for the first time. And we walked through the concrete migration path from ZooKeeper mode through the 3.9 bridge release to a clean Kafka 4.0 KRaft cluster.
The direction is clear: Kafka is simplifying, scaling, and broadening its reach as a messaging platform. If you’re still on ZooKeeper mode, now is the time to plan your path forward.
