A Trillion-Events-a-Day Ingestion Pipeline: Claim Check, Rebatching and the Cost Playbook
Record first, refine second, route third. How object storage plus pointer messages, deduplication, zstd and a shadow pipeline cut event-pipeline cost.
On this page
Analytics platforms live on a simple promise: send us events and we will not lose them. Keeping that promise at 20 times last year’s volume while staying cheap is an architecture problem, and Statsig has described a clean way to think about it.
Three stages, three single priorities
| Stage | One priority | Everything else is secondary |
|---|---|---|
| Request recorder | Never lose data | Latency, format, cleverness |
| Log processing | Refine and verify | Speed |
| Routing | Deliver to the right destination efficiently | Uniformity across customers |
Giving each stage exactly one priority is a useful design discipline. The recorder is deliberately dumb and durable. The expensive logic lives later, where a bug cannot lose the original data.
Stage 1: the recorder
- Client SDKs through a load balancer
Requests spread across servers so none is overwhelmed.
- Authenticate, then queue internally
Only valid, authorised batches are accepted. They go onto an in-memory queue so brief processing delays do not drop data.
- Rebatch
Many small batches are combined into large ones, sized by dynamic configuration, and written as compressed files to object storage. Fewer, bigger writes cost less and compress better.
- Persist with fallbacks
A semi-permanent layer with several fallbacks, across regions and even cloud platforms. Pods carry enough memory to ride out roughly an hour of downstream trouble without losing or slowing intake.
Stage 2: the queue carries pointers
Do not put event data on the message bus. Write the data to object storage, and publish only a small message saying where it is. The bus stays fast and cheap, consumers fetch the bytes on demand, and you avoid per-message size limits.
A serverless pub/sub topic carries just file pointers, while zstd-compressed batches sit in object storage. Putting the event bodies themselves into the managed bus would be prohibitively expensive at this volume, and the serverless choice removes servers to run.
Stage 3: business logic and routing
Processing. Rebatch again for processing efficiency, with an at-least-once guarantee: if a step fails it is retried, never skipped. Remove duplicates with a fast cache such as Memcached, as early as possible, so duplicates never cost downstream work. Allow teams to attach plugins for customer-specific logic such as extra tags or restructured events, so one pipeline serves many needs instead of one pipeline per customer.
Routing. A router sends data to destinations by customer preference, event type and priority:
The shadow pipeline in detail
A release candidate runs in parallel with the same code path as production, but its output goes nowhere customers can see.
- Send a burst of a few million events through it to mimic peak load.
- Compare its outputs with production’s through automated checks, looking for missing or wrong data outside tolerance.
- Watch scaling behaviour, including horizontal pod autoscaling, under the burst.
- Promote only if it matches.
This is how a small team ships changes to a trillion-event system without a stomach ache.
The cost playbook
| Lever | What changed | Why it saves |
|---|---|---|
| Offload bulk data to object storage | Only pointers go through the managed bus | Object storage is far cheaper per byte than messaging |
| Spot capacity for async work | Non-urgent jobs run on discounted, interruptible machines | Retries make interruption harmless |
| Early deduplication | Drop duplicates near the front | Downstream stages process less |
| zstd instead of zlib | Better compression at lower CPU cost | Smaller storage and less compute |
| Bigger batches, bigger recorder | Recorder moved from 2 to 12 CPUs per instance to build larger batches | Fewer storage writes per event |
| Load jobs vs streaming | Bulk-load anything not needed in seconds | Streaming is priced for latency |
| Size by host, not by pod | Tune to actual host utilisation, no strict per-pod limits | Idle reserved headroom is wasted money |
| Aggressive packing | Stack many pods per host, tune concurrency to avoid contention | Fewer machines |
Do
Classify every destination as latency-sensitive or not, measure cost per million events by stage, make each stage re-runnable from the one before, and rehearse a downstream outage to confirm intake survives it.
Don't
Stream everything in real time because it is the default. Most analytics data is looked at hours later, and you are paying a premium to move it faster than anyone reads it.
Takeaway
At extreme volume, pipeline design is a series of separations: durable recording from clever processing, bulk bytes from the small messages that point at them, urgent data from patient data, and production from the release candidate. Each separation lets you spend on speed only where speed is valued, and that is where the cost goes.
Frequently asked questions
What is the claim check pattern?
Instead of putting a large payload on a message queue, store the payload in cheap object storage and send a small message containing a pointer to it. Consumers fetch the data when they process the message. It cuts queue cost and size limits, and keeps queues fast.
Why rebatch events into larger files?
Per-request and per-write costs dominate at high volume, and small writes waste throughput. Combining many small client batches into large compressed files reduces storage operations, improves compression and lets downstream jobs process big units, at the cost of a little latency.
When should a pipeline stream into a warehouse and when use load jobs?
Stream only data that someone needs within seconds, because streaming ingestion is priced and rate-limited for low latency. Everything else should be loaded in bulk batches, which are far cheaper. Splitting the two by latency need is one of the biggest cost levers.
What is a shadow pipeline?
A copy of the production pipeline running a release candidate on mirrored or synthetic traffic, with its outputs compared against production's. It reveals data differences, performance regressions and scaling problems before the new version touches real customers.

/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.