---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 4 min
---

A Trillion-Events-a-Day Ingestion Pipeline: Claim Check, Rebatching and the Cost Playbook

Record first, refine second, route third. How object storage plus pointer messages, deduplication, zstd and a shadow pipeline cut event-pipeline cost.

– –

Analytics platforms live on a simple promise: send us events and we will not lose them. Keeping that promise at 20 times last year’s volume while staying cheap is an architecture problem, and Statsig has described a clean way to think about it.

Three stages, three single priorities

StageOne priorityEverything else is secondary
Request recorderNever lose dataLatency, format, cleverness
Log processingRefine and verifySpeed
RoutingDeliver to the right destination efficientlyUniformity across customers

Giving each stage exactly one priority is a useful design discipline. The recorder is deliberately dumb and durable. The expensive logic lives later, where a bug cannot lose the original data.

Stage 1: the recorder

  1. Client SDKs through a load balancer

    Requests spread across servers so none is overwhelmed.

  2. Authenticate, then queue internally

    Only valid, authorised batches are accepted. They go onto an in-memory queue so brief processing delays do not drop data.

  3. Rebatch

    Many small batches are combined into large ones, sized by dynamic configuration, and written as compressed files to object storage. Fewer, bigger writes cost less and compress better.

  4. Persist with fallbacks

    A semi-permanent layer with several fallbacks, across regions and even cloud platforms. Pods carry enough memory to ride out roughly an hour of downstream trouble without losing or slowing intake.

Stage 2: the queue carries pointers

The claim check pattern

Do not put event data on the message bus. Write the data to object storage, and publish only a small message saying where it is. The bus stays fast and cheap, consumers fetch the bytes on demand, and you avoid per-message size limits.

A serverless pub/sub topic carries just file pointers, while zstd-compressed batches sit in object storage. Putting the event bodies themselves into the managed bus would be prohibitively expensive at this volume, and the serverless choice removes servers to run.

Stage 3: business logic and routing

Processing. Rebatch again for processing efficiency, with an at-least-once guarantee: if a step fails it is retried, never skipped. Remove duplicates with a fast cache such as Memcached, as early as possible, so duplicates never cost downstream work. Allow teams to attach plugins for customer-specific logic such as extra tags or restructured events, so one pipeline serves many needs instead of one pipeline per customer.

Routing. A router sends data to destinations by customer preference, event type and priority:

Warehouse router the cost lever
Latency-insensitive data goes through bulk load jobs, much cheaper than streaming. Time-critical data uses the low-latency write API at higher cost.
Side-effects service integrations
Event-level triggers: webhooks to third parties, workflows in external tools.
Real-time stream live views
A fast in-memory store backs live dashboards and monitoring with near-instant reads.
Shadow pipeline safe releases
Runs the release candidate alongside production without touching customers.
Advertisement

The shadow pipeline in detail

A release candidate runs in parallel with the same code path as production, but its output goes nowhere customers can see.

  1. Send a burst of a few million events through it to mimic peak load.
  2. Compare its outputs with production’s through automated checks, looking for missing or wrong data outside tolerance.
  3. Watch scaling behaviour, including horizontal pod autoscaling, under the burst.
  4. Promote only if it matches.

This is how a small team ships changes to a trillion-event system without a stomach ache.

The cost playbook

LeverWhat changedWhy it saves
Offload bulk data to object storageOnly pointers go through the managed busObject storage is far cheaper per byte than messaging
Spot capacity for async workNon-urgent jobs run on discounted, interruptible machinesRetries make interruption harmless
Early deduplicationDrop duplicates near the frontDownstream stages process less
zstd instead of zlibBetter compression at lower CPU costSmaller storage and less compute
Bigger batches, bigger recorderRecorder moved from 2 to 12 CPUs per instance to build larger batchesFewer storage writes per event
Load jobs vs streamingBulk-load anything not needed in secondsStreaming is priced for latency
Size by host, not by podTune to actual host utilisation, no strict per-pod limitsIdle reserved headroom is wasted money
Aggressive packingStack many pods per host, tune concurrency to avoid contentionFewer machines

Do

Classify every destination as latency-sensitive or not, measure cost per million events by stage, make each stage re-runnable from the one before, and rehearse a downstream outage to confirm intake survives it.

Don't

Stream everything in real time because it is the default. Most analytics data is looked at hours later, and you are paying a premium to move it faster than anyone reads it.

Takeaway

At extreme volume, pipeline design is a series of separations: durable recording from clever processing, bulk bytes from the small messages that point at them, urgent data from patient data, and production from the release candidate. Each separation lets you spend on speed only where speed is valued, and that is where the cost goes.

Advertisement

Frequently asked questions

What is the claim check pattern?

Instead of putting a large payload on a message queue, store the payload in cheap object storage and send a small message containing a pointer to it. Consumers fetch the data when they process the message. It cuts queue cost and size limits, and keeps queues fast.

Why rebatch events into larger files?

Per-request and per-write costs dominate at high volume, and small writes waste throughput. Combining many small client batches into large compressed files reduces storage operations, improves compression and lets downstream jobs process big units, at the cost of a little latency.

When should a pipeline stream into a warehouse and when use load jobs?

Stream only data that someone needs within seconds, because streaming ingestion is priced and rate-limited for low latency. Everything else should be loaded in bulk batches, which are far cheaper. Splitting the two by latency need is one of the biggest cost levers.

What is a shadow pipeline?

A copy of the production pipeline running a release candidate on mirrored or synthetic traffic, with its outputs compared against production's. It reveals data differences, performance regressions and scaling problems before the new version touches real customers.

Sources & further reading

/* Comments */