Apache Kafka, the leading platform for distributed event streaming, is undergoing a significant architectural shift. The platform is phasing out Apache ZooKeeper, its long-standing metadata management tool, in favor of an internally developed alternative. This move is set to simplify Kafka’s architecture, improve scalability, and streamline cluster management for users.
ZooKeeper has historically been used to store Kafka’s cluster metadata, manage dynamic configurations, and oversee topic partitions. However, as Colin McCabe, a member of Kafka’s project management committee and an engineer at Confluent, explains, ZooKeeper introduces an additional layer of complexity. Transitioning to an internal metadata management system will allow Kafka to handle these tasks directly, providing stronger guarantees for features like versioning while reducing operational overhead.
The replacement, Kafka Raft (KRaft), introduces a protocol designed for internally managed metadata. With KRaft, Kafka’s metadata will be stored in a distributed log, eliminating the need for a separate ZooKeeper system. This change promises enhanced scalability, a critical improvement for managing large Kafka deployments. Additionally, by consolidating metadata management within Kafka itself, the platform becomes more straightforward to configure and maintain.
The timeline for fully decommissioning ZooKeeper is still under discussion. A vote on the schedule is expected soon, with plans to declare KRaft generally available in Kafka 3.3, targeted for release in August. This release will support both ZooKeeper and KRaft, providing users with a transitional period. ZooKeeper is expected to be deprecated in the release following Kafka 3.3 and completely removed in Kafka 4.0. Although an end-of-life date for ZooKeeper has not been finalized, McCabe noted that KRaft is moving into production “very soon,” marking a significant evolution in the Kafka ecosystem and a major step forward for the platform.