Counting at Global Scale: Best-Effort vs Eventually Consistent Counters
A counter looks trivial until it spans regions and retries. Five designs compared, and a hybrid: log increments, roll up in immutable windows, cache reads.
On this page
Counters are where “simple” systems go to die. A view counter, an experiment tally, a usage meter for billing: all are +1, and all fail differently. The first design decision is not how to count, it is how wrong you are allowed to be.
Two tiers of correctness
| Best-effort | Eventually consistent | |
|---|---|---|
| Priority | Speed and cost | Accuracy and durability |
| Typical use | A/B test counts, rate approximations, transient metrics | Billing, compliance, usage reports |
| Loses or double-counts? | Possible | Not in the final value |
| Retention | Short, with a TTL | Long |
| Stale reads | Acceptable | Acceptable, but converging to exact |
Both tiers share the platform requirements: high availability, high throughput and horizontal scale across regions, including spikes on launch day.
A small, safe API
POST add_and_get_count
{
"namespace": "user_metrics",
"counter_name": "views",
"delta": 5,
"idempotency_token": { "token": "evt-8f3a", "generation_time": "2026-11-14T10:00:00Z" }
}- AddAndGetCount: apply a positive or negative delta, return the new value.
- GetCount: read, possibly slightly stale for speed.
- ClearCount: reset, also idempotent.
The idempotency token is the important field. Without it, any network retry silently inflates the count.
Five ways to build one, and where each breaks
5. Event log of increments. Store every increment as an event with event_time and event_id. The pair doubles as the idempotency key, and totals can always be recomputed. The costs: storage grows without bound, reading means scanning all events for a counter, and wide partitions in wide-column stores slow queries.
The hybrid: log, roll up, cache
Each option fixes one thing and breaks another, so the production design combines three of them.
- Log every increment in a time-series store
Record
event_time, a uniqueevent_idand the counter key. Group events into time buckets (a minute or an hour) so no partition grows unboundedly. Unique ids make retries harmless. - Roll up in the background over immutable windows
A rollup process sums events in windows that can no longer change, then writes the total and a last rollup timestamp to durable storage. The next rollup starts from that checkpoint, so cost scales with new events, not total history.
- Serve reads from a cache
The cache holds the last rolled-up count and its timestamp. A read returns instantly, and if the value is stale it triggers a background rollup for next time.
If you aggregate a window that can still receive late events, your stored total is silently wrong. Only roll up what is closed. The count lags by a small, bounded delay, and in exchange it never drifts.
Scaling the rollup pipeline
Rollups are triggered by tiny rollup events that say “this counter needs aggregating”, with no data in them. Design points:
- In-memory queues per rollup server, giving parallelism without a durable queue to run. A fast non-cryptographic hash (such as xxHash) routes the same counter to the same queue so work is not duplicated.
- Deduplicate within a queue. Many triggers for one counter collapse into a set, so it rolls up once per window.
- Batch dynamically with back-pressure. Process counters in batches, and slow intake when the store or workers are saturated.
- Accept the risk, bound it. If a rollup server crashes, queued triggers vanish, but frequently used counters get re-triggered by the next read or write, and the event log still holds the truth. Overlap during deployments is safe, because windows are immutable.
- Scale by adding queues and redeploying servers that drain gracefully.
Low-cardinality counters converge almost immediately. Millions of high-cardinality counters converge as batches work through, so tune window size and batch settings against how fresh the data must be.
Choosing quickly
| Need | Use |
|---|---|
| Rough experiment tallies, short life | Cache counter with TTL |
| Billing, compliance, durable totals | Hybrid event log with rollups |
| Unique counts (not totals) | A probabilistic sketch such as HyperLogLog |
| Counters merged across regions with no coordination | A CRDT counter |
| One server’s metrics | In-process counters flushed to a metrics system |
Do
Decide the tier per metric, always include an idempotency key on durable counts, keep raw events long enough to recompute, and test crash scenarios by killing a rollup server mid-window.
Don't
Use a cache counter for money, or one hot database row for a counter that every request touches. Both work at demo scale and fail loudly at real scale.
Takeaway
Counting at scale is a choice about error, not arithmetic. Best-effort counters trade accuracy for speed and cost, and durable ones keep every increment, deduplicate by id, roll up closed windows into checkpoints and serve cached results. Use the cheapest tier each metric can tolerate, and make retries harmless from the start.
Frequently asked questions
Why is a distributed counter hard?
Increments arrive from many machines and regions at once, retries can double-count, instances can crash before saving, and a single hot row becomes a contention point. You must choose between speed, accuracy and durability, and a different choice suits A/B test tallies than billing totals.
What is the difference between best-effort and eventually consistent counting?
Best-effort counting favours speed and low cost, using in-memory caches, accepting lost or duplicated increments. Eventually consistent counting favours accuracy and durability, logging increments and aggregating them, so the total converges to the correct value shortly after the events. Use best-effort for experiments, eventual for money.
How do idempotency tokens prevent double counting?
Each increment carries a unique id. The system records ids it has applied, or uses the id as a key so a repeated write overwrites itself. If a client retries after a timeout, the duplicate has the same id and is ignored, so the count is incremented once.
Why aggregate over immutable time windows?
Once a window is closed and no new events can land in it, its total never changes. Rolling up only finalised windows lets you store a checkpoint and add later windows to it, instead of rescanning all events on every read, with no risk of missing late arrivals inside a window.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.