Scaling Three Ways: Out, Up and the Team, and the Cache Lease That Protects the Database
Add machines, make each one cheaper, and keep the team shipping. Plus how a lease token stops herds and stale writes, the fix for read-through caches.
On this page
Instagram’s early scaling story is a useful template because it is so ordinary: a few servers added by hand before weekends, an overloaded database, an engineer’s time eaten by operations. The fixes fall on three separate axes, and teams who only work one of them stall.
The early pain, and what it signals
| Symptom | What it meant |
|---|---|
| Engineers add servers manually before each weekend | Capacity planning by hand, no autoscaling |
| Slow responses and occasional outages | Too few instances, unbalanced traffic |
| Relational database overloaded | Reads not cached, queries not tuned, replicas managed manually |
| Constant manual monitoring | No alerting that acts for you |
| Scaling work delaying features | The team is the bottleneck |
Scale, for context: hundreds of millions of daily users, around 100 million uploads and several billion likes a day, each like a database write.
Axis 1: scale out
Spread work across more hardware.
- Move to a larger infrastructure with multiple data centres, with load balancing across them.
- Replace one database with a distributed setup: PostgreSQL with leader-follower replication for relational data and Cassandra for feeds and activity logs.
- Use specialised storage for media instead of a general file system, so serving an image takes minimal disk operations, fronted by a CDN.
- Migrate incrementally, copying and synchronising data in both places instead of a big-bang cutover.
Axis 2: scale up
Adding servers is not sustainable if each one is wasteful. Make every machine do more.
The request path for a “like”
- The web tier takes the request
A Django view validates and writes the like to PostgreSQL.
- Counts come from the cache
Display counts are read from Memcached instead of querying the database each time.
- Everything slow is queued
A task for the notification goes to a message broker (RabbitMQ), so the user’s tap returns immediately.
- A worker finishes the job
A Celery worker picks it up, builds the notification and sends it, with automatic retries if it fails.
Strong consistency goes where it matters (authentication, relationships and anything financial) on PostgreSQL. Eventual consistency goes where a delay is harmless (feeds, activity) on Cassandra. The cache is invalidated on writes.
The read-path problem: the cache miss
A cache in front of a database works until a hot key is missing, then two ugly things happen.
Thundering herd. A popular key expires or is invalidated. Thousands of requests miss at the same instant and all query the database for the same row.
Stale set. Slower and sneakier:
1. Reader A misses the cache for key K
2. A reads K from the database (value v1)
3. Writer updates the database to v2 and deletes K from the cache
4. A, finally, writes v1 into the cache
-> the cache now holds stale v1 until it expires
The lease mechanism
A lease is a token issued by the cache on a miss.
- First miss gets a lease
Instead of an empty answer, the cache returns a lease token to the first client to miss. That client is now responsible for fetching from the database.
- Others wait or use stale
Other clients that miss the same key are told to wait a short time and retry, or are handed a recently-expired value flagged as stale if the application accepts it. Only one database query runs.
- Fill with the token
The lease holder queries the database and stores the result with its token. The cache accepts the store only if the token is still valid.
- Writes invalidate tokens
When a writer deletes the key, any outstanding lease for it is voided. A reader who then tries to store a stale value is rejected, which closes the stale-set race.
The same token that limits who may refill a key also proves the refill is still current. Rate limiting the herd and rejecting late stale writes are the same idea.
Axis 3: scale the team
More engineers on one codebase means more conflicts, more regressions and harder debugging. The remedy is process, not heroics:
- Continuous deployment in many small releases per day, each easy to understand and revert.
- Gradual rollout: canary servers first, then a growing share of users.
- Real-time monitoring that catches performance regressions after each change.
- Code review and automated tests as the gate, not release-day ceremonies.
A scaling checklist
| Axis | Ask |
|---|---|
| Out | Can I add a node without code changes? Is the data tier distributed? |
| Up | What are the top five costs per request? Which can be batched, cached or removed? |
| Team | How many deploys per day? How fast can a bad change be rolled back? |
| Cache | What happens when the hottest key disappears? Can a stale value be written back? |
Do
Alternate between out and up: scale out to buy time, scale up to make the next doubling cheaper, and protect the database with leases or an equivalent single-flight mechanism on every read-through cache.
Don't
Solve every slowdown by adding servers. Each server also multiplies your database connections and cache misses, so waste scales with the fleet.
Takeaway
Scale is three problems: capacity, efficiency and people. Add machines for capacity, tune what each machine does for efficiency, and release in small safe steps so the team can keep growing. Then close the gap every cache leaves open, with leases that allow only one refill at a time and refuse stale ones.
Frequently asked questions
What is the difference between scaling out and scaling up?
Scaling out adds more machines and spreads work across them. Scaling up makes each machine do more work, by profiling, batching queries, caching and rewriting hot paths. Out buys capacity, up buys efficiency. Doing only out means paying for waste, doing only up hits the limit of one machine.
What is a cache lease?
A token a cache hands to the first client that misses on a key, giving it the right to fetch from the database and fill the cache. Other clients that miss are told to wait briefly or use a slightly stale value. It prevents a thundering herd of identical database queries.
How can a cache end up holding stale data after a race?
A reader misses, reads the old value from the database, then a writer updates the database and deletes the cache key, then the reader writes its old value into the cache. A lease token is invalidated by the delete, so the reader's late fill is rejected.
Why does deployment speed count as scaling?
As engineer count grows, large infrequent releases produce merge conflicts, risky changes and slow debugging. Many small releases behind canaries and quick rollback keep risk per change low, so the organisation can keep adding people without slowing down.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.