How LinkedIn Replaced ZooKeeper With Kafka and xDS Service Discovery at Massive Scale
Inside LinkedIn's Next-Gen Service Discovery: replacing legacy ZooKeeper clusters with Apache Kafka streams and an Envoy xDS control plane.
On this page
At LinkedIn, delivering a seamless experience across the feed, messaging, and job applications depends on the coordination of tens of thousands of microservices running across hundreds of thousands of container instances worldwide.
For over a decade, the backbone connecting this distributed fleet was a custom service discovery platform powered by Apache ZooKeeper and the Rest.li RPC framework.
However, as LinkedIn scaled past 100,000 distinct service instances handling billions of daily calls, ZooKeeper hit fundamental architectural limits. High instance churn triggered thundering herd watcher storms, global data center cross-talk caused session drops, and tight coupling to Java libraries blocked multi-language container modernization.
To overcome these barriers, LinkedIn engineered Next-Gen Service Discovery: completely decommissioning ZooKeeper in favor of an Apache Kafka event-driven control plane combined with Envoy-compatible xDS APIs.
+-------------------------------------------------------------------------------+
| LINKEDIN SERVICE DISCOVERY EVOLUTION |
+-------------------------------------------------------------------------------+
Legacy ZooKeeper Architecture (2012-2024):
[Service Instance] ---> Ephemeral Node (/d2/uris/...) ---> [ZooKeeper Quorum]
|
[Client App (Java Only)] <--- Long-Lived Watcher Storms ----------+
* Bottlenecks: Watcher stampedes, single-threaded ZK commit queue, Java-only.
Next-Gen xDS + Kafka Architecture:
[Service Instance / K8s Pod]
|
v
[Registration Service] ---> [Apache Kafka Service Changelog Topic]
|
v
[Aggregator & Translation Control Plane]
|
v
[Envoy-Standard xDS gRPC Streaming API]
|
+--------------------------------+--------------------------------+
| |
v v
[Java Rest.li Client Fleet] [Envoy Sidecar / Mesh]
1. The Anatomy of ZooKeeper’s Failure at Scale
In the legacy architecture, service discovery was built on LinkedIn’s internal “Dynamic Discovery” (D2) framework:
- When a service instance booted, it established a TCP session with a regional ZooKeeper cluster and wrote an ephemeral znode under
/d2/uris/<service_name>/<instance_id>. - Client services placed watchers on the service’s znode directory.
- If an instance died, its TCP heartbeat ceased, ZooKeeper deleted the ephemeral znode, and fired watch notifications to all subscribing clients.
The Three Failure Modes of ZooKeeper at Scale
THE ZOOKEEPER THUNDERING HERD
[100 Pods Deploy / Restart Simultaneously]
|
v
[ZooKeeper Leader: Processes 100 Ephemeral Node Deletes]
|
+--- Fires Watchers to 5,000 Subscribed Client Pods!
|
v
[5,000 Clients Simulataneously Re-Query ZooKeeper: "GET /d2/uris/feed"]
|
v
[ZooKeeper Inbound Queue Overflows -> Dropped Heartbeats -> Cascading Outage!]
- Read Stampedes (Thundering Herd): ZooKeeper’s watch mechanism is an edge-trigger notification (“something changed, come read the directory”). When a large service deployed 500 pods, 5,000 dependent clients simultaneously queried ZooKeeper to re-read the full cluster directory, overwhelming ZooKeeper’s memory bus.
- Single-Threaded Commit Pipeline: While ZooKeeper handles reads across followers, all state mutations (znode registrations and deletions) must pass through a single-threaded Paxos leader commit pipeline. Under high churn, registration latency jumped from milliseconds to tens of seconds.
- The Java Monoculture: The D2 client logic (consistent hashing, load balancing, health tracking, and ZooKeeper watcher management) was embedded inside a massive Java library. Non-Java microservices (Python, Go, Node.js) were second-class citizens unable to integrate cleanly.
2. Next-Gen Architecture: Decoupling via Apache Kafka
LinkedIn recognized that service discovery updates are fundamentally an append-only event stream:
Event: Instance 10.2.4.15:8080 added to service SearchEvent: Instance 10.2.4.19:8080 health check degradedEvent: Instance 10.2.4.12:8080 terminated
Rather than using a consensus coordination database like ZooKeeper as a pub-sub pipe, LinkedIn shifted registration event streams directly into Apache Kafka.
+-------------------------------------------------------------------------------+
| NEXT-GEN REGISTRATION PIPELINE |
+-------------------------------------------------------------------------------+
[Microservice Pod (Any Language)]
|
v
[Local Registration Agent / Kubernetes Controller]
|
v
[Registration Gateway Service]
|
v
[Apache Kafka: service-registry-events (Partitioned by Service Name)]
|
v
[Discovery Aggregator Engine (In-Memory Topology State Machine)]
|
v
[Delta Translation: Compiles State into Envoy xDS Resource Formats]
Why Kafka Excels for Discovery Events
- Immense Throughput: Kafka easily absorbs tens of thousands of registration events per second without head-of-line blocking.
- Log Compaction: The registry topic uses Kafka log compaction, retaining only the latest state for each instance key while preserving a complete audit log of cluster churn.
- Backpressure Isolation: If a consumer slows down, Kafka buffers events safely on disk without threatening the publisher’s stability.
3. The Data Plane: Adopting Envoy’s xDS Protocol
To break free from Java library dependencies, LinkedIn decoupled the discovery control plane from client runtimes by adopting the open xDS (Discovery Service) API protocol, originally popularized by Envoy.
Instead of writing bespoke client libraries for every programming language, LinkedIn microservices consume standard xDS gRPC streaming endpoints:
- LDS (Listener Discovery Service): Pushes networking port configurations.
- RDS (Route Discovery Service): Pushes path-based routing rules and weights.
- CDS (Cluster Discovery Service): Pushes upstream service definitions.
- EDS (Endpoint Discovery Service): Pushes the active IP and port lists for backend instances.
[Discovery Aggregator]
|
| xDS gRPC Streaming (HTTP/2 Multiplexed Delta Updates)
|
+---> [Envoy Service Mesh Sidecars (C++, Non-Java Services)]
|
\---> [xDS-Native Rest.li Java Clients (In-Process)]
Delta xDS: Slashed Network Bandwidth
Legacy systems pushed the entire membership list whenever a single server changed. If a cluster had 2,000 instances, losing one pod transmitted a 50KB JSON payload to every listener.
LinkedIn implemented Delta xDS (Incremental EDS). The control plane transmits only the exact IP address that was added or removed. Network bandwidth consumption dropped by over 85%.
4. Multi-Region Failover and Data Center Isolation
ZooKeeper clusters often spanned multiple regional data centers, exposing service discovery to cross-continental WAN latency spikes and network partitions.
Next-Gen Service Discovery enforces regional control plane autonomy:
+-----------------------------------------------------------------------+
| REGIONAL CONTROL PLANE ISOLATION |
+-----------------------------------------------------------------------+
Data Center 1 (East) Data Center 2 (West)
+-------------------------------+ +-------------------------------+
| Local Registration & Kafka | | Local Registration & Kafka |
| | | | | |
| v | | v |
| Local xDS Aggregator Engine | | Local xDS Aggregator Engine |
| | | | | |
| v | | v |
| Local Microservice Fleet | | Local Microservice Fleet |
+-------------------------------+ +-------------------------------+
^ ^
| ===== MirrorMaker 2 (Cross-DC Mirroring) == |
Each data center runs its own local Kafka cluster and its own xDS aggregator fleet. Discovery within a data center has zero external WAN dependencies. Cross-region routing data is replicated asynchronously via Kafka MirrorMaker, guaranteeing that a fiber cut across the Atlantic never stalls local request routing.
Architectural Comparison Matrix
| Dimension | Legacy ZooKeeper Architecture | Next-Gen Kafka + xDS Architecture |
|---|---|---|
| State Storage | Hierarchical ephemeral znodes in RAM | Event-driven compacted Kafka topics |
| Update Notification | Edge-triggered watchers (Thundering herd) | Streamed push deltas over gRPC (xDS) |
| Language Support | Heavy Java library coupling | Polyglot (Envoy sidecars, Go, Python, Java) |
| Bandwidth Profile | Full membership state pushed on any change | Incremental Delta EDS (Only the changed instance) |
| Failure Domain | Global ZK quorum dependencies | Fully autonomous regional control planes |
| Scale Ceiling | ~100,000 instances before watcher exhaustion | Millions of instances with linear horizontal scaling |
Summary and Key Takeaways
- Decouple consensus from pub-sub: ZooKeeper is an outstanding consensus coordinator for leader election, but a terrible high-throughput pub-sub pipe for volatile instance churn.
- Treat service registration as an event log: Utilizing Apache Kafka for service discovery state captures registrations as an append-only, compacted event stream.
- Standardize on xDS protocols: Adopting Envoy-compatible xDS APIs eliminates monolithic language-specific discovery libraries and empowers polyglot container adoption.
- Push deltas, not full state: Incremental Delta EDS transmits only the specific instance added or removed, slashing cross-service network overhead by 85%.
- Isolate regional control planes: Design service discovery so that individual data centers operate autonomously without synchronous cross-region WAN dependencies.

/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.