Replacing ZooKeeper with internally managed metadata will improve scalability and management, say Kafka developers.
Change is coming for users of Apache Kafka, the leading distributed event-streaming platform. Apache ZooKeeper, Kafka’s metadata management tool, will soon be phased out in favor of internal technology. This significant shift is set to improve the platform’s scalability and management capabilities.
Colin McCabe, a member of the Apache Kafka project management committee and an engineer at Confluent, a company leveraging Kafka, explained the rationale behind the change. ZooKeeper has traditionally been used to store persistent cluster metadata, manage dynamic configurations, and handle topics and their partitions. However, this adds an extra layer of management complexity. By storing metadata internally within Kafka, the management process will be simplified and will enable better guarantees around issues such as versioning, McCabe said.
The new protocol, Kafka Raft (KRaft), will replace ZooKeeper. KRaft will manage metadata internally by storing it in a distributed log. According to McCabe, the primary benefit of this transition will be enhanced scalability, though management will also see improvements. Users will no longer need to deploy and maintain a separate system for managing Kafka clusters.
While the exact timeline for ZooKeeper’s decommissioning is not yet determined, a vote on the schedule is expected in the next week or two. The current proposal suggests making KRaft generally available in the upcoming Kafka 3.3 release, slated for August. ZooKeeper would be deprecated in the release following that and completely removed by Kafka 4.0. In Kafka 3.3, both ZooKeeper and KRaft will be available, offering users a transitional period.
KRaft mode has been available since Kafka 2.8, released in April 2021, but it has not been production-ready. Kafka 3.3 will be the first version to offer a production-ready KRaft mode. Currently, Kafka developers are focused on achieving full feature parity between KRaft and ZooKeeper, which is said to be nearly complete. McCabe assures that developers will not face a steep learning curve when transitioning to KRaft, as the same APIs are supported. However, operators might have to learn new management practices. “Our hope is that new administrators will find this easier and that existing administrators will find it manageable to transition,” McCabe said.
Confluent sees the deprecation of ZooKeeper as a crucial advancement for the Kafka community. Jun Rao, Kafka’s co-creator and co-founder of Confluent, stated that this move simplifies deployment and operations while improving scalability by a factor of 10 due to more efficient metadata handling. Rao expressed optimism about the community’s work on this transition, anticipating that KRaft will soon be production-ready.
Kafka is extensively used, with more than 80% of Fortune 100 companies relying on it, according to the Apache Software Foundation. Accessible from the Apache Kafka website, the platform is a distributed system consisting of clients and servers that communicate via a high-performance TCP protocol. Kafka can be deployed on virtual machines, bare-metal hardware, and containers, both on-premises and in cloud environments.
The transition to KRaft represents a significant milestone in Kafka’s evolution, promising to enhance its robustness and ease of use for the vast number of organizations that depend on it.