---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 5 min
---

Scaling Three Ways: Out, Up and the Team, and the Cache Lease That Protects the Database

Add machines, make each one cheaper, and keep the team shipping. Plus how a lease token stops herds and stale writes, the fix for read-through caches.

– –

Instagram’s early scaling story is a useful template because it is so ordinary: a few servers added by hand before weekends, an overloaded database, an engineer’s time eaten by operations. The fixes fall on three separate axes, and teams who only work one of them stall.

The early pain, and what it signals

SymptomWhat it meant
Engineers add servers manually before each weekendCapacity planning by hand, no autoscaling
Slow responses and occasional outagesToo few instances, unbalanced traffic
Relational database overloadedReads not cached, queries not tuned, replicas managed manually
Constant manual monitoringNo alerting that acts for you
Scaling work delaying featuresThe team is the bottleneck

Scale, for context: hundreds of millions of daily users, around 100 million uploads and several billion likes a day, each like a database write.

Axis 1: scale out

Spread work across more hardware.

  • Move to a larger infrastructure with multiple data centres, with load balancing across them.
  • Replace one database with a distributed setup: PostgreSQL with leader-follower replication for relational data and Cassandra for feeds and activity logs.
  • Use specialised storage for media instead of a general file system, so serving an image takes minimal disk operations, fronted by a CDN.
  • Migrate incrementally, copying and synchronising data in both places instead of a big-bang cutover.

Axis 2: scale up

Adding servers is not sustainable if each one is wasteful. Make every machine do more.

Fewer, batched queries database relief
A feed load that issues many tiny queries becomes a few batched ones. Remove redundant reads.
Cache hot data memory is cheaper than queries
Memcached in front of the databases for profiles, posts and counts.
Profile, then rewrite the hot path measure first
Find the few functions that burn CPU, and move them to a faster implementation, such as C or C++ extensions for hot Python code.
Use memory better density
Reduce per-process memory so more workers fit per host.

The request path for a “like”

  1. The web tier takes the request

    A Django view validates and writes the like to PostgreSQL.

  2. Counts come from the cache

    Display counts are read from Memcached instead of querying the database each time.

  3. Everything slow is queued

    A task for the notification goes to a message broker (RabbitMQ), so the user’s tap returns immediately.

  4. A worker finishes the job

    A Celery worker picks it up, builds the notification and sends it, with automatic retries if it fails.

Strong consistency goes where it matters (authentication, relationships and anything financial) on PostgreSQL. Eventual consistency goes where a delay is harmless (feeds, activity) on Cassandra. The cache is invalidated on writes.

Advertisement

The read-path problem: the cache miss

A cache in front of a database works until a hot key is missing, then two ugly things happen.

Thundering herd. A popular key expires or is invalidated. Thousands of requests miss at the same instant and all query the database for the same row.

Stale set. Slower and sneakier:

1. Reader A misses the cache for key K
2. A reads K from the database (value v1)
3. Writer updates the database to v2 and deletes K from the cache
4. A, finally, writes v1 into the cache
   -> the cache now holds stale v1 until it expires

The lease mechanism

A lease is a token issued by the cache on a miss.

  1. First miss gets a lease

    Instead of an empty answer, the cache returns a lease token to the first client to miss. That client is now responsible for fetching from the database.

  2. Others wait or use stale

    Other clients that miss the same key are told to wait a short time and retry, or are handed a recently-expired value flagged as stale if the application accepts it. Only one database query runs.

  3. Fill with the token

    The lease holder queries the database and stores the result with its token. The cache accepts the store only if the token is still valid.

  4. Writes invalidate tokens

    When a writer deletes the key, any outstanding lease for it is voided. A reader who then tries to store a stale value is rejected, which closes the stale-set race.

One mechanism, two bugs fixed

The same token that limits who may refill a key also proves the refill is still current. Rate limiting the herd and rejecting late stale writes are the same idea.

Axis 3: scale the team

More engineers on one codebase means more conflicts, more regressions and harder debugging. The remedy is process, not heroics:

  • Continuous deployment in many small releases per day, each easy to understand and revert.
  • Gradual rollout: canary servers first, then a growing share of users.
  • Real-time monitoring that catches performance regressions after each change.
  • Code review and automated tests as the gate, not release-day ceremonies.

A scaling checklist

AxisAsk
OutCan I add a node without code changes? Is the data tier distributed?
UpWhat are the top five costs per request? Which can be batched, cached or removed?
TeamHow many deploys per day? How fast can a bad change be rolled back?
CacheWhat happens when the hottest key disappears? Can a stale value be written back?

Do

Alternate between out and up: scale out to buy time, scale up to make the next doubling cheaper, and protect the database with leases or an equivalent single-flight mechanism on every read-through cache.

Don't

Solve every slowdown by adding servers. Each server also multiplies your database connections and cache misses, so waste scales with the fleet.

Takeaway

Scale is three problems: capacity, efficiency and people. Add machines for capacity, tune what each machine does for efficiency, and release in small safe steps so the team can keep growing. Then close the gap every cache leaves open, with leases that allow only one refill at a time and refuse stale ones.

Advertisement

Frequently asked questions

What is the difference between scaling out and scaling up?

Scaling out adds more machines and spreads work across them. Scaling up makes each machine do more work, by profiling, batching queries, caching and rewriting hot paths. Out buys capacity, up buys efficiency. Doing only out means paying for waste, doing only up hits the limit of one machine.

What is a cache lease?

A token a cache hands to the first client that misses on a key, giving it the right to fetch from the database and fill the cache. Other clients that miss are told to wait briefly or use a slightly stale value. It prevents a thundering herd of identical database queries.

How can a cache end up holding stale data after a race?

A reader misses, reads the old value from the database, then a writer updates the database and deletes the cache key, then the reader writes its old value into the cache. A lease token is invalidated by the delete, so the reader's late fill is rejected.

Why does deployment speed count as scaling?

As engineer count grows, large infrequent releases produce merge conflicts, risky changes and slow debugging. Many small releases behind canaries and quick rollback keep risk per change low, so the organisation can keep adding people without slowing down.

Sources & further reading

/* Comments */