---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 5 min
---

Kafka Consumers at Scale: Rebalance Storms, Poison Pills and the Proxy Escape Hatch

Why consumer groups stall, how one bad message blocks a partition, and how a delivery proxy decouples consumer scale from partition count.

– –

Kafka at small scale is a pleasure. At the scale where tens of thousands of consumers across clouds read a trillion messages a day, the same design shows three sharp edges. Each one has a cheap fix at modest scale and an architectural fix at large scale.

Problem 1: rebalancing

A consumer group divides a topic’s partitions among its members. Whenever membership looks different, Kafka redistributes them, and consumption stalls while it does.

TriggerTypical cause
A consumer joins or leavesDeploys, rolling restarts, autoscaling in Kubernetes
Heartbeats missed beyond the session timeoutProcess crash, long stop-the-world GC pause, node trouble
Gap between polls exceeds the max poll intervalProcessing a batch took too long, so the broker assumes the consumer is stuck

The third is the sneakiest: the consumer is healthy and busy, but slow batches look like death, causing a rebalance that makes everything slower.

Keep polling fast separate poll from work
Poll in one thread and hand records to a worker pool through a bounded queue, so the poll loop never waits on slow processing. Pause the partition if the queue fills (backpressure).
Tune the timeouts session and poll settings
Size the session timeout and max poll interval to your real processing times, and reduce max poll records so a batch finishes inside the window.
Use cooperative rebalancing less disruption
Incremental rebalancing moves only the partitions that must move, instead of revoking everything, so most consumers keep working through a rebalance.
Static membership restarts without churn
Giving each consumer a stable instance id lets a quick restart rejoin without triggering a full reassignment.

Problem 2: the poison pill

A poison pill is a message the consumer cannot process, ever: malformed JSON, a missing required field, a value that violates a business rule, or a bug triggered by one rare shape of input.

The naive loop is the trap:

poll -> process fails -> do not commit -> poll returns same batch -> fails again -> ...

One bad record blocks every message behind it in that partition, which is head-of-line blocking. Lag climbs, alerts fire, and nothing is actually down.

  1. Bound the retries

    Retry transient failures a few times with backoff. A parse error will not fix itself, so distinguish retryable errors (timeout) from permanent ones (invalid payload).

  2. Park the failure

    Send the message to a dead letter queue with the error, source topic, partition and offset, then commit past it so the partition moves on.

  3. Alert and replay

    Monitor the DLQ’s size. Fix the bug or the data, then replay the parked messages. Never let the DLQ become a silent bin.

  4. Validate early

    Check schema at the producer, with a schema registry and compatibility rules, so malformed data rarely reaches the topic at all.

Commit in order, carefully

If workers finish out of order, you can only commit the highest contiguous offset. If offsets 1, 2, 3 and 5 are done but 4 is not, commit 3. Committing 5 would lose 4 on a crash.

Advertisement

Problem 3: partition count is a cost lever

Parallelism within a group is capped by partitions. Ten partitions mean ten useful consumers, and the eleventh sits idle. To go faster you must add partitions, and that has costs:

  • Brokers have practical limits on partitions per broker. Past them you need bigger brokers even if throughput does not require it.
  • Changing partition count needs coordination between the Kafka team, producers and consumers, and changes how keys map to partitions.
  • More partitions mean more file handles, memory and threads on the brokers.

For one team that is acceptable. With thousands of pipelines, it is a tax on every one.

The architectural escape: a delivery proxy

Walmart’s engineers have described putting a Messaging Proxy Service between Kafka and consumer applications. It turns Kafka’s pull model into HTTP delivery so worker scale no longer depends on partitions.

ComponentJob
Reader threadOne thread polls Kafka and writes into a bounded queue. If the queue fills, it pauses (backpressure)
Pending queueBuffer so reading and delivering run at their own speeds
Order iteratorEnsures at most one in-flight message per key, preserving per-key order
Writer threadsPOST each message to a stateless consumer service, retry a few times, then route to a DLQ
Offset commit threadPeriodically commits the highest contiguous processed offset per partition

Consumer services receive plain HTTP requests, are stateless, and scale in Kubernetes with message volume. They can use specific response codes to signal “this is a poison pill” so the proxy parks it right away. Built as a Kafka Connect sink, it inherits multi-tenancy, DLQ handling and offset management.

The results reported: most rebalances disappeared because the reader always hands records off in time, poison pills are handled cleanly, and Kafka clusters can be sized by throughput rather than by consumer parallelism.

The honest costs

  • Another service to build, deploy, monitor and keep available.
  • HTTP delivery adds latency compared with in-process consumption, and the proxy itself must handle its own rebalances.
  • Per-key ordering limits parallelism for hot keys, exactly as partitioning would.

Choosing a level of fix

Your situationDo this
Occasional rebalances, a few consumersTune timeouts, poll size, cooperative rebalancing, static membership
Slow per-message workWorker pool behind a bounded queue, pause on backpressure
Bad messages stall a partitionBounded retry, dead letter queue, schema validation
Hundreds of teams, consumer scale pinned to partitionsConsider a proxy or shared consumer platform

Do

Make processing idempotent, bound every retry, give every failed message a destination with context, alert on consumer lag and DLQ depth, and test a rebalance during a deploy before production does.

Don't

Retry a failing record forever, commit offsets before processing is done, or add partitions as the first answer to slow consumers.

Takeaway

Consumer groups fail in predictable ways: rebalances from membership churn or slow polling, poison pills from unbounded retries, and scaling limits from partition coupling. Fix the first two with timeouts, backpressure, cooperative rebalancing and a dead letter queue. When the third becomes expensive across an organisation, decouple workers from partitions with a proxy, accepting one more component in exchange for freedom.

Advertisement

Frequently asked questions

What triggers a Kafka consumer rebalance?

A consumer joining or leaving the group (deploys, scaling, crashes), the broker missing heartbeats within the session timeout (often a long garbage-collection pause), or a consumer taking longer than the maximum poll interval between polls because processing a batch ran slow. Each one pauses consumption while partitions are reassigned.

What is a poison pill message?

A message that makes the consumer fail every time it is processed, because of malformed data, unexpected values or a bug. If the consumer retries the same batch forever, that single message blocks all later messages in its partition, a head-of-line blocking problem.

What is a dead letter queue?

A separate topic or store where messages that cannot be processed after bounded retries are parked with error details. The main flow keeps moving, and the failed messages can be inspected, fixed and replayed later instead of blocking the partition or being silently dropped.

Why can't I just add consumers to go faster?

Within a group, a partition is assigned to at most one consumer, so consumers beyond the partition count sit idle. Raising throughput by adding consumers means adding partitions, which has cluster-wide costs, or decoupling consumption from partitions with a worker pool or delivery proxy.

Sources & further reading

/* Comments */