Caches That Survive Change: Warming, CDC Invalidation and Partitioning Apart From the Database

The usual cache lesson is “it makes reads fast”. The operational lesson is the reverse: a cache you cannot resize, replace or fail over without a latency spike is a liability. The database is sized assuming the cache absorbs most reads, so a cold cache is a database outage waiting for a trigger.

Why change is dangerous

The naive growth plan is to build a bigger empty copy, write to both, wait for old entries to expire, then switch. It works for short-lived data and fails for entries that never expire or rarely change, since they never repopulate. It also means paying for two systems during the wait.

Technique 1: warm from a healthy copy

Instead of waiting for traffic to refill the cache, copy the data across. A well-designed warmer has three roles, loosely coupled.

1. Controller: orchestrate

   Picks the source replica (typically the one with the most data), creates a queue for communication, starts the dump on source nodes, launches the populators, watches progress and cleans up afterward.

2. Dumper: extract

   Runs next to each cache node. Lists keys (memcached’s LRU crawler can enumerate them), saves keys and values into chunk files, uploads each chunk to object storage when it reaches a size, and posts a message with the chunk’s location.

3. Populator: fill

   Pulls messages from the queue, downloads each chunk, and inserts into the new nodes. It starts as soon as the first chunk exists rather than waiting for the dump to end, and can scale with the backlog.

Two design choices carry the safety:

Warming a single node is the same pipeline with a filter. When a node restarts empty and signals the controller, the controller does not use that node’s own replica as the source, because its peers kept receiving writes while it was down. It dumps from other replicas, and only for keys that hash to the node being warmed, which makes the job small and fast. Netflix has reported replacing a node and refilling it in under fifteen minutes this way, and warming new replicas holding about twelve terabytes in roughly two hours.

Technique 2: invalidate from the change log

The classic cache bug is a write path that forgets to clear a key. Every new service, script or manual fix is another chance. The robust approach moves invalidation out of application code.

write -> database -> change log (binlog / WAL)
                            |
                     CDC service tails it
                            |
              delete or refresh matching cache keys

An integrated cache works like this: the query layer, not each microservice, reads from the cache on a hit and falls through to storage on a miss. A separate change-data-capture service tails the database’s log and invalidates the matching entries. Every writer is covered, because the database log sees every write.

Caveats to design for:

• Lag. Invalidation arrives after the commit, so a reader can briefly see an old value. Set a TTL as a backstop and decide which reads must bypass the cache.
• Ordering. Apply changes in log order, and prefer delete-then-refill over writing values from the log, which risks storing an older version.
• Races. A reader who missed just before a write can repopulate the cache with the old value after the invalidation. Version your entries or use short TTLs on misses.

Technique 3: warm across regions

A region failover sends traffic to a cache that has never seen those keys. One mitigation is to replicate keys continuously: tail the cache’s write stream in one region, and in the other region have a consumer read each key through the normal query path, which fills the local cache from the local database. You pay a little extra read traffic all the time to avoid a cliff during failover.

Technique 4: partition the cache differently

Suppose cache and database are sharded identically, so cache cluster 3 fronts database shard 3. If cluster 3 goes down, every miss lands on shard 3, which was sized for a cache-protected load. One cache failure becomes one database hot spot.

Partition the cache with a different scheme and each cache cluster holds keys from many database shards. Losing a cluster spreads its misses thinly across all shards, each absorbing a small increase.

same scheme:       cache C3 down  -> 100% of its misses -> DB shard 3   (overload)
different scheme:  cache C3 down  -> 1/N of misses      -> each shard   (absorbed)

Checklist

Takeaway

A cache is only as good as its worst day: resize, replacement, failover and invalidation. Warm from healthy replicas with a decoupled pipeline and add-only writes, let the database log drive invalidation, replicate hot keys across regions, and shard the cache independently from the data. Then the cache is something you can operate, not just something you depend on.