---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 5 min
---

Operating Kafka at 1.8 Trillion Events a Day: Auditing, ACLs and Lag-Aware Load Balancing

Disk-first logging, clusters split by use case, end-to-end audit counts, self-service ACLs, and balancing uneven consumers with lag instead of round robin.

– –

The Kafka tutorials stop where the real work starts. Once hundreds of teams produce events, the questions change: who may write to what, how do you prove nothing was lost, and why does one slow consumer force you to double the fleet?

Step 1: make producing events boring

Applications should not each embed Kafka client tuning, retries and serialisation choices. A two-step logging architecture removes that burden:

  1. The client library writes to local disk

    It handles file rotation and locations, offers a small producer API, and enforces serialisation standards. The application never blocks on the network.

  2. A forwarder daemon ships to Kafka

    It reads the files, sends events, tracks offsets in the files and deletes finished ones. The platform team can change batching, configuration and Kafka versions without any application redeploy.

The trade is latency. Agoda reported a 99th percentile around ten seconds for analytics workloads, which is fine for analytics and unacceptable for others. Time-sensitive paths needing sub-second delivery bypass the file stage and write to Kafka directly. Offer both, and be explicit about which use case gets which.

Step 2: split clusters by use case

One giant cluster per data centre is simple until one workload misbehaves. Several smaller clusters, each dedicated to a use case or group of them, give:

BenefitHow
Blast-radius containmentA runaway producer or broker failure affects one cluster, not everyone
Tuning per workloadReal-time clusters use short retention and high network throughput. Replication clusters keep data longer
Safer upgradesRoll out a version to the least critical cluster first

Also run ZooKeeper (or the metadata quorum) on dedicated nodes, separate from brokers. It holds cluster metadata and coordinates leader election, so you do not want it competing with brokers for disk and CPU.

Step 3: monitor, then audit

Monitoring answers “is it running”: broker throughput, partition counts, consumer lag, request latencies, collected from broker metrics endpoints into a time-series store and graphed.

Auditing answers a harder question: “did every event arrive?”

producer lib:  count per 1-minute bucket -> audit topic (separate cluster)
consumer/sink: count per 1-minute bucket -> audit topic
compare:       bucket 12:04  produced 1,000,000  landed 999,712  -> 288 missing

Counting runs in a background thread inside the client library, aggregated by time bucket and written to a separate cluster so audit traffic cannot disturb the data path. The counts feed a near real-time analytics platform and long-term storage. When a bucket mismatches you know where and when to look, and you can alert on completeness the way you alert on latency.

Add security before you need it

Early Kafka use for low-stakes telemetry often runs unauthenticated. Then usage grows, a misbehaving client degrades a cluster, and nobody can say whose it is. Agoda added authentication and authorisation in 2021. It is far cheaper to ship identity from the start.

Step 4: identity, ACLs and self-service

A workable access model has five parts: authentication with SASL, ACLs for per-topic and per-group permissions, credential generation, credential assignment to teams and users, and a self-service portal so teams request access without queuing at the platform team. Identity also gives you quotas per client, so one heavy user cannot starve the rest.

Advertisement

Step 5: balancing uneven consumers

Kafka’s default producer partitioning (round robin or sticky) and consumer assignors (range, round robin) assume all consumers are equal and all messages cost the same. Real fleets are neither: pods land on different hardware generations, and some messages trigger a third-party API call or database query while others do not.

A worked example: two fast workers handle 20 messages per second each, one slow worker handles 10. Total capacity is 50 per second. With equal shares and a 50 per second input, each gets about 16.7. The fast ones coast, the slow one falls behind forever. To fix it with equal shares you would need five machines instead of three, paying for two idle ones.

Static weights do not help when message cost varies unpredictably, so the fix is dynamic and uses lag as the signal.

Lag-aware producer steer the input
The producer caches per-partition lag, refreshed periodically so it does not hammer the brokers, and sends fewer messages to partitions with high lag and more to those with low lag. Given lags of 110, 150, 80, 400, 120 and 380, it would steer away from the 400 and 380 partitions. When lag is even, distribution is even.
Lag-aware consumer steer the assignment
Works even when several groups read the same topic, where a producer cannot optimise for all of them. A consumer far behind can voluntarily leave, triggering a rebalance in which a custom assignor reassigns partitions by each member’s lag and capacity.

The cost of frequent rebalancing is made tolerable by incremental cooperative rebalancing (Kafka 2.4 onward), which moves only the partitions that change owners instead of stopping the whole group.

Applying it at normal size

Do thisWhen
Local-file or sidecar forwardingMany teams, many languages, you want to own Kafka behaviour centrally
Clusters per workload classYour first cluster contains both critical and best-effort traffic
Audit counts per time bucketAnyone downstream asks “is the data complete?”
ACLs and per-client quotasMore than one team shares a cluster
Lag-based balancingConsumers differ in speed and you are over-provisioning to hide it

Do

Treat Kafka as a product with users: give them a simple producer path, access they can request themselves, and a completeness guarantee backed by numbers.

Don't

Over-provision the consumer fleet to mask uneven load. Scale hides the imbalance, and the bill pays for it every month.

Takeaway

At scale, Kafka succeeds on platform decisions: events written safely to disk first, clusters isolated by workload, counts that prove completeness, identity and ACLs from the start, and load balancing that follows lag instead of assuming equal workers. Adopt them in that order as the number of teams grows.

Advertisement

Frequently asked questions

Why write events to local disk before sending them to Kafka?

A local file decouples applications from Kafka's availability and lets a separate forwarder handle batching, retries and upgrades. Teams stop owning Kafka client behaviour. The cost is added latency, so sub-second use cases should write to Kafka directly instead.

Why split Kafka into several clusters by use case?

Isolation. A noisy or failing workload cannot take down others, and each cluster can be tuned for its job: short retention and high throughput for real-time processing, long retention for replication. The price is more clusters to monitor, upgrade and secure.

How do you audit that no Kafka messages were lost?

Count messages at each stage, by time bucket, and publish those counts to a separate audit topic or cluster. Compare producer counts with consumer and sink counts per bucket. A mismatch pinpoints where messages were lost, duplicated or delayed.

Why is round-robin partitioning a problem for uneven consumers?

Round robin gives every partition and every consumer the same share regardless of capacity. If one consumer runs on slower hardware or gets costlier messages, its share outgrows its speed and lag builds, forcing you to over-provision the whole group to compensate for the slowest member.

Sources & further reading

/* Comments */