---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 5 min
---

Five Million Checks a Second Under 5 ms: Three Generations of an Enforcement Path

From a database to cache-aside, full in-memory copies, Bloom filters, streaming sync and off-heap data: how a safety check on every request got fast.

– –

Hot-path lookups are where good architectures get decided. This one sits on the way to almost every action a member takes, so each design choice shows up as latency for everyone and as a safety hole if it is wrong.

What the system does

An abuse-prevention platform combines machine learning models that spot unusual behaviour, rule-based systems for explicit policy, human review for borderline cases, and multi-faceted restrictions: temporary limits (no connection requests for a while) up to permanent blocks. All of that decision-making produces one thing the rest of the product consumes: a restriction record that must be enforced instantly everywhere.

Generation one: from a database to a pile of caches

  1. Relational tables

    Restrictions in tables, one per restriction type, with CRUD workflows. It stops working when a microservice fleet makes high query volumes and low latency demands.

  2. Server-side cache-aside

    Check an in-memory cache first, fall back to the database on a miss and fill the cache asynchronously, with a TTL. It cut database load, but the caches were per-host and struggled under heavy hit demand.

  3. Client-side caching

    A library caches restriction data on each calling application’s hosts, removing a network hop. The cost was consistency between client and server caches, and database strain when caches reloaded.

  4. Full refresh-ahead cache

    Each client holds all restriction data in memory and polls for changes. Reads never miss and never touch the network, so latency drops sharply. The costs: large memory per host, heavy reload on every restart or deploy, and database spikes during refreshes.

  5. Bloom filters

    Replace the full dataset with a compact probabilistic structure that answers whether a record exists. Memory falls and queries are fast. The trade is false positives: the filter can say “restricted” for someone who is not.

Mind the direction of the error

A Bloom filter never says “absent” for something present, but it sometimes says “present” for something absent. For enforcement that means occasional wrongful flags. Use it as a prefilter, then confirm positives against authoritative data before taking action on a member.

Generation two: a streaming-fed store

Goals set the bar: 4 to 5 million queries per second, under 5 ms latency, 99.999 percent availability, horizontal scale, and fresh data.

ChoiceWhy
Move records to a distributed NoSQL document storeA relational database struggled with the throughput and latency
Emit every new record to a streaming platformConsumers on many servers and data centres stay in sync in near real time
Look up in memoryNo repeated database queries on the hot path

It brought a new pain: bootstrapping. A restarted server had to reload all records from the database, taking over 30 minutes for big datasets and straining the store while doing so. Growth and adversarial spikes also tested limits.

The CAP choice

The team prioritised consistency and availability over partition tolerance. Their reasoning: stale or wrong restriction data is a security and experience failure, availability is essential, and past partitioned systems introduced latency that violated the five-millisecond target. In practice, that means keeping the data and its consumers close, within a controlled topology, and using streaming to hold replicas in sync.

Advertisement

Generation three: memory and bootstrap

Off-heap storage tame the collector
Keeping huge datasets in the managed heap makes garbage collection long and spiky. Storing the bulk in memory outside the heap, in compact binary layouts, leaves the collector far less to scan, so pauses and latency spikes shrink and more data fits.
Eager cache on a derived-data platform fast bootstrap
An eagerly populated local cache backed by a data platform that serves bulk snapshots, instead of a slow record-by-record reload. Restarts and new instances become quick.
Compact data structures bitset-like
Dense representations for membership and flags cut memory and make lookups cheap.
Resilience to spikes adversarial load
Attackers create bursts of records. The design absorbs them without slow reloads or heap pressure.

Five principles

  1. Move data to the request, not the request to the data. Hot-path decisions belong in memory next to the caller.
  2. Pick the error direction on purpose. Probabilistic structures are fine when you know which kind of wrong you are choosing, and confirm before acting.
  3. Design for startup, not only steady state. A cache you cannot rebuild quickly is a liability on every deploy.
  4. Stream changes, do not poll. Event-driven sync keeps data fresh without pummelling the database.
  5. Plan for hostile traffic. Enforcement systems are targets, so test the spike where an attacker, not a user, shapes the load.

What to copy at a smaller size

Your scaleDo this
Hundreds of requests per secondA cache-aside lookup with a short TTL is plenty
Thousands per second, small datasetLoad the whole set in memory and subscribe to changes
Large dataset, tight memoryBloom filter prefilter plus authoritative confirmation
Latency spikes from garbage collectionProfile first, then consider off-heap or compact structures
Slow restartsSnapshot-based bootstrap plus a change stream, not row-by-row loading

Do

Measure p99 latency and restart time as first-class numbers, keep an authoritative path behind every approximation, and load-test with attack-shaped traffic.

Don't

Add another cache layer to fix a problem caused by the previous cache layer. Each layer you add multiplies consistency questions.

Takeaway

The enforcement path improved by moving closer to the caller, then fixing what that move broke: memory, restarts, garbage collection and spikes. The final shape is an eager in-memory store fed by a change stream, bootstrapped from snapshots, compact in memory and honest about the error rate of any approximation it uses.

Advertisement

Frequently asked questions

Why must an enforcement check be so fast?

It runs on every request across every product: loading a feed, sending a message, requesting a connection. A few milliseconds on each call multiplies across millions of requests per second. The check has to be in-process or very close, and highly available, because failing open or closed both have costs.

What is the trade-off of using a Bloom filter for restrictions?

A Bloom filter answers 'definitely not present' or 'probably present'. It saves large amounts of memory, but false positives can flag members who are not restricted. A common design uses it as a fast first pass and confirms positives against the real data before enforcing.

What is a refresh-ahead cache?

A cache that proactively reloads data before it is requested, often holding the full dataset in memory and polling or streaming for changes. Reads never miss, so latency is minimal, at the cost of memory per client and heavy work on startup.

Why move data off-heap in a Java service?

Large heaps of long-lived objects make garbage collection slow and spiky. Storing the bulk data in memory outside the managed heap, in compact binary structures, shrinks what the collector scans, so pauses and latency spikes drop.

Sources & further reading

/* Comments */