How Reddit Migrated a Petabyte Kafka Fleet From EC2 to Kubernetes With Zero Downtime
How Reddit migrated a 500-broker, petabyte-scale Apache Kafka fleet from AWS EC2 to Kubernetes and Strimzi with zero downtime and no client changes.
On this page
At Reddit, Apache Kafka is the central nervous system connecting hundreds of microservices.
Every post, upvote, comment, direct message, advertising impression, and moderation signal traverses an enormous Kafka deployment comprising over 500 brokers handling petabytes of live data.
For years, this infrastructure ran on bare AWS EC2 instances managed via Puppet and custom Python deployment scripts.
While reliable in early stages, the EC2 operating model hit an operational wall:
- Painful Node Lifecycle Upgrades: Upgrading Linux kernels or provisioning replacement brokers required hours of manual SSH scripting and disk rebalancing.
- Hardware Underutilization: Brokers were over-provisioned on expensive
i3enstorage-optimized EC2 instances that sat idle during non-peak traffic windows. - Operational Fragmentation: The rest of Reddit’s microservices fleet had migrated to Kubernetes, leaving Kafka as an isolated legacy infrastructure silo.
Reddit committed to an audacious infrastructure migration: move the entire petabyte-scale Kafka fleet from EC2 into Kubernetes using the open-source Strimzi operator, with zero downtime, zero data loss, and zero code changes across hundreds of client microservices.
+-------------------------------------------------------------------------------+
| REDDIT KAFKA MIGRATION ARCHITECTURE |
+-------------------------------------------------------------------------------+
[Upstream Microservice Clients (Unaware of Migration)]
|
v (Connects via Static Internal DNS: broker-101.kafka.reddit.com)
+-------------------------------------------------------------------------------+
| THE DNS FACADE LAYER |
+-------------------------------------------------------------------------------+
| Phase 1: DNS points to EC2 Host: 10.20.4.15 |
| Phase 2: DNS migrates to K8s NLB / NodePort: 10.40.8.20 |
+-------------------------------------------------------------------------------+
| |
v v
[Legacy EC2 Kafka Cluster] [Next-Gen Kubernetes Strimzi Cluster]
Broker 101, 102, 103 (EC2 VMs) Broker 101, 102, 103 (K8s Pods + EBS gp3)
| ^
\======= Inter-Broker Mirroring & Replication ==/
1. The Four Non-Negotiable Migration Constraints
Reddit established four unyielding architectural constraints before touching a single production broker:
- Zero Downtime: Reddit’s real-time feeds, chat, and ads could not tolerate even 5 minutes of maintenance downtime.
- Zero Client Changes: Requiring hundreds of engineering teams to update bootstrap servers, certificates, or client SDK configurations was an organizational impossibility.
- No Loss of Message Ordering or Delivery Guarantees: State cannot be duplicated or dropped during migration handoffs.
- Instant, Zero-Impact Reversibility: If a Kubernetes cluster degraded under peak load, traffic had to roll back to EC2 in minutes without data loss.
2. The Migration Strategy: The Hybrid Expanded Cluster
A common migration anti-pattern is deploying an entirely new destination cluster and mirroring data using tools like MirrorMaker 2.
At petabyte scale, cross-cluster mirroring introduces dual-writing complexities, offset translation drift, and double the infrastructure footprint.
Reddit adopted the Hybrid Cluster Expansion Strategy:
+-------------------------------------------------------------------------------+
| THE HYBRID EXPANDED CLUSTER |
+-------------------------------------------------------------------------------+
[Single Shared ZooKeeper / KRaft Coordination Quorum]
|
+-------------------------------+-------------------------------+
| |
v v
[Active EC2 Brokers (1..500)] [New K8s Strimzi Pods (1..500)]
Cluster ID: "reddit-prod-kafka" Cluster ID: "reddit-prod-kafka"
- Both EC2 instances and Kubernetes pods belong to the EXACT SAME Kafka cluster!
- Brokers replicate partitions natively across the internal network fabric!
Instead of running two separate clusters, Reddit joined the new Kubernetes pods directly into the existing EC2 Kafka cluster.
The Kafka cluster temporarily doubled in size. To Kafka, a broker pod running in Kubernetes looked identical to an EC2 instance.
3. The DNS Facade and the Broker ID Swap
How did Reddit migrate clients from EC2 instances to Kubernetes pods without updating client connection strings?
They used a DNS Facade paired with 1:1 Broker ID Swaps:
THE 1:1 BROKER MIGRATION SEQUENCE
Step 1: Deploy K8s Pod with temporary Broker ID 601.
Replicate partition data from EC2 Broker 101 to K8s Pod 601.
Step 2: Once In-Sync Replicas (ISR) are 100% caught up:
- Terminate EC2 Broker 101.
- Re-label K8s Pod 601 as Broker ID 101!
Step 3: Update Internal Route53 DNS Record:
"broker-101.kafka.reddit.com" CNAME updated:
FROM: ec2-10-20-4-15.compute.internal
TO: k8s-nlb-10-40-8-20.elb.us-east-1.amazonaws.com
Result: Client microservices reconnect automatically during TCP keep-alive
refreshes, landing on the Kubernetes pod with zero disruption!
By keeping broker IDs and DNS names identical, client connection pools seamlessly re-established TCP sockets onto Kubernetes without client-side reconfiguration.
4. Forking Strimzi: Solving the Identity Dilemma
The open-source Strimzi Kafka Operator is designed to manage Kubernetes-native Kafka clusters declaratively.
By default, Strimzi assigns broker IDs sequentially starting from 0 (broker-0, broker-1, etc.).
Reddit’s production EC2 cluster used non-contiguous, historical broker IDs spanning hundreds of integers (broker-101, broker-245, broker-409).
Running stock Strimzi would have forced re-indexing all broker IDs, violating the zero-client-change invariant.
Reddit forked Strimzi to support:
- Custom, Static Broker ID Mappings: Allowing specific pods to claim arbitrary existing broker IDs.
- External Advertised Listeners: Coordinating static AWS Network Load Balancer (NLB) IPs and Route53 DNS records during pod migrations.
Production Results
| Migration Metric | Pre-Migration (EC2) | Post-Migration (Kubernetes) |
|---|---|---|
| Fleet Scale | 500+ EC2 bare-metal instances | Managed Kubernetes Pods (EKS) |
| Storage Engine | Fixed local instance NVMe | Elastic AWS EBS gp3 volumes |
| Infrastructure Cost | High (Over-provisioned idle VMs) | 35% reduction in cloud compute spend |
| Provisioning Time | 2 to 4 hours per broker | < 5 minutes via Strimzi declarative CRDs |
| Migration Outage Time | N/A | 0 seconds (Zero downtime achieved) |
Summary and Key Takeaways
- Avoid dual-cluster migrations when possible: Joining target Kubernetes pods directly into the existing cluster allows native Kafka replication to handle data synchronization.
- Decouple clients using a DNS facade: Route client traffic through persistent internal DNS hostnames to migrate underlying physical nodes without client reconfigurations.
- Execute 1:1 broker replacements: Match broker IDs and replica partition assignments sequentially to preserve cluster topology.
- Fork operators when enterprise constraints require it: If open-source Kubernetes operators impose restrictive conventions (like sequential broker IDs), patch them to preserve production invariants.
- Declarative infrastructure slashes maintenance toil: Shifting Kafka from imperative Puppet scripts to Kubernetes Strimzi operators cuts broker recovery times from hours to minutes.

/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.