What triggers a Kafka consumer rebalance?
A consumer joining or leaving the group (deploys, scaling, crashes), the broker missing heartbeats within the session timeout (often a long garbage-collection pause), or a consumer taking longer than the maximum poll interval between polls because processing a batch ran slow. Each one pauses consumption while partitions are reassigned.
Answered in
Kafka Consumers at Scale: Rebalance Storms, Poison Pills and the Proxy Escape HatchWhy consumer groups stall, how one bad message blocks a partition, and how a delivery proxy decouples consumer scale from partition count.
Read the full analysisOther questions this article answers
More system design & architecture questions
- How much spare capacity should a gateway keep?
- Why are bounded queues important for availability?
- Should a load balancer always move requests away from an unhealthy server?
- Can a circuit breaker prevent every cascading failure?
- Why does pagination improve API performance beyond just 'smaller responses'?
- Why is asynchronous logging faster than writing logs synchronously?
- When does caching actually help API performance, and when doesn't it?
- Why does payload compression trade CPU time for network time, and when is that worth it?
Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the System Design & Architecture beat.