Five Million Checks a Second Under 5 ms: Three Generations of an Enforcement Path
From a database to cache-aside, full in-memory copies, Bloom filters, streaming sync and off-heap data: how a safety check on every request got fast.
On this page
Hot-path lookups are where good architectures get decided. This one sits on the way to almost every action a member takes, so each design choice shows up as latency for everyone and as a safety hole if it is wrong.
What the system does
An abuse-prevention platform combines machine learning models that spot unusual behaviour, rule-based systems for explicit policy, human review for borderline cases, and multi-faceted restrictions: temporary limits (no connection requests for a while) up to permanent blocks. All of that decision-making produces one thing the rest of the product consumes: a restriction record that must be enforced instantly everywhere.
Generation one: from a database to a pile of caches
- Relational tables
Restrictions in tables, one per restriction type, with CRUD workflows. It stops working when a microservice fleet makes high query volumes and low latency demands.
- Server-side cache-aside
Check an in-memory cache first, fall back to the database on a miss and fill the cache asynchronously, with a TTL. It cut database load, but the caches were per-host and struggled under heavy hit demand.
- Client-side caching
A library caches restriction data on each calling application’s hosts, removing a network hop. The cost was consistency between client and server caches, and database strain when caches reloaded.
- Full refresh-ahead cache
Each client holds all restriction data in memory and polls for changes. Reads never miss and never touch the network, so latency drops sharply. The costs: large memory per host, heavy reload on every restart or deploy, and database spikes during refreshes.
- Bloom filters
Replace the full dataset with a compact probabilistic structure that answers whether a record exists. Memory falls and queries are fast. The trade is false positives: the filter can say “restricted” for someone who is not.
A Bloom filter never says “absent” for something present, but it sometimes says “present” for something absent. For enforcement that means occasional wrongful flags. Use it as a prefilter, then confirm positives against authoritative data before taking action on a member.
Generation two: a streaming-fed store
Goals set the bar: 4 to 5 million queries per second, under 5 ms latency, 99.999 percent availability, horizontal scale, and fresh data.
| Choice | Why |
|---|---|
| Move records to a distributed NoSQL document store | A relational database struggled with the throughput and latency |
| Emit every new record to a streaming platform | Consumers on many servers and data centres stay in sync in near real time |
| Look up in memory | No repeated database queries on the hot path |
It brought a new pain: bootstrapping. A restarted server had to reload all records from the database, taking over 30 minutes for big datasets and straining the store while doing so. Growth and adversarial spikes also tested limits.
The CAP choice
The team prioritised consistency and availability over partition tolerance. Their reasoning: stale or wrong restriction data is a security and experience failure, availability is essential, and past partitioned systems introduced latency that violated the five-millisecond target. In practice, that means keeping the data and its consumers close, within a controlled topology, and using streaming to hold replicas in sync.
Generation three: memory and bootstrap
Five principles
- Move data to the request, not the request to the data. Hot-path decisions belong in memory next to the caller.
- Pick the error direction on purpose. Probabilistic structures are fine when you know which kind of wrong you are choosing, and confirm before acting.
- Design for startup, not only steady state. A cache you cannot rebuild quickly is a liability on every deploy.
- Stream changes, do not poll. Event-driven sync keeps data fresh without pummelling the database.
- Plan for hostile traffic. Enforcement systems are targets, so test the spike where an attacker, not a user, shapes the load.
What to copy at a smaller size
| Your scale | Do this |
|---|---|
| Hundreds of requests per second | A cache-aside lookup with a short TTL is plenty |
| Thousands per second, small dataset | Load the whole set in memory and subscribe to changes |
| Large dataset, tight memory | Bloom filter prefilter plus authoritative confirmation |
| Latency spikes from garbage collection | Profile first, then consider off-heap or compact structures |
| Slow restarts | Snapshot-based bootstrap plus a change stream, not row-by-row loading |
Do
Measure p99 latency and restart time as first-class numbers, keep an authoritative path behind every approximation, and load-test with attack-shaped traffic.
Don't
Add another cache layer to fix a problem caused by the previous cache layer. Each layer you add multiplies consistency questions.
Takeaway
The enforcement path improved by moving closer to the caller, then fixing what that move broke: memory, restarts, garbage collection and spikes. The final shape is an eager in-memory store fed by a change stream, bootstrapped from snapshots, compact in memory and honest about the error rate of any approximation it uses.
Frequently asked questions
Why must an enforcement check be so fast?
It runs on every request across every product: loading a feed, sending a message, requesting a connection. A few milliseconds on each call multiplies across millions of requests per second. The check has to be in-process or very close, and highly available, because failing open or closed both have costs.
What is the trade-off of using a Bloom filter for restrictions?
A Bloom filter answers 'definitely not present' or 'probably present'. It saves large amounts of memory, but false positives can flag members who are not restricted. A common design uses it as a fast first pass and confirms positives against the real data before enforcing.
What is a refresh-ahead cache?
A cache that proactively reloads data before it is requested, often holding the full dataset in memory and polling or streaming for changes. Reads never miss, so latency is minimal, at the cost of memory per client and heavy work on startup.
Why move data off-heap in a Java service?
Large heaps of long-lived objects make garbage collection slow and spiky. Storing the bulk data in memory outside the managed heap, in compact binary structures, shrinks what the collector scans, so pauses and latency spikes drop.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.