Scaling the API Layer: Stateless Servers, Rate Limits, Caching and Async Work
The API tier is the easy part to scale, if you keep it stateless. The practices that matter: limits, pagination, caching, queues and graceful overload.
On this page
Adding API servers is the easy part of scaling. The hard part is making sure the extra servers do not all queue behind the same slow database, and that one enthusiastic client cannot starve everyone else.
What the layer is responsible for
Process requests against a contract, authenticate and authorise callers, orchestrate calls to backend services and aggregate results, then format and return responses. Every one of those steps is a place for latency to accumulate, and every request exercises all of them.
1. Stay stateless
Any instance must be able to serve any request. That means:
- Sessions in a shared store or in signed tokens, not in instance memory.
- Uploads streamed to object storage, not held on local disk.
- Configuration from the environment, not from files edited on one box.
Then scaling is mechanical: add instances, let the load balancer spread them, remove them when traffic falls.
2. Make the cheap path cheap
3. Cache what repeats
| Layer | What | Notes |
|---|---|---|
| Client and CDN | Public, repeated GETs with Cache-Control and ETag | Free scale. Revalidate with conditional requests |
| Shared cache (Redis) | Computed results, lookups, rate-limit counters | Use cache-aside: read cache, on miss load and store |
| In-process | Tiny, hot, rarely changing data | Per-instance copies can disagree briefly |
Set an expiry on everything, decide how a write invalidates the entry, and never cache user-specific data under a key that is not user-specific.
When a popular cache entry expires, hundreds of requests can miss at once and hit the database together. Let one request refresh while others serve the slightly stale value, and add random jitter to expiry times so entries do not all expire together.
4. Move slow work off the request path
If something takes seconds, depends on a flaky third party, or can be retried, do not make the caller wait on a thread.
POST /reports -> 202 Accepted, Location: /reports/42
(job placed on a queue)
GET /reports/42 -> { "status": "running" } ... { "status": "done", "url": ... }
A queue also levels load: spikes pile up in the queue and workers drain it at a steady rate, so the database behind them sees smooth demand. Make jobs idempotent, because queues deliver at least once.
5. Rate limit for fairness and survival
Limits stop one client from consuming everyone’s capacity and protect you from runaway scripts and retry storms. Common shapes:
| Limiter | Protects against |
|---|---|
| Per-client request rate (token bucket) | A single caller flooding the API |
| Concurrency limit | Too many slow requests in flight at once |
| Load shedder | Total demand beyond what the fleet can serve |
| Priority tiers | Letting critical traffic through while dropping the optional |
Return 429 with Retry-After and tell clients how to back off. The limiter itself needs shared state, usually a fast store, so all instances enforce the same limit.
6. Degrade, do not collapse
When demand exceeds capacity, queueing everything makes every request slow, then every request times out. Instead shed load: reject the excess quickly with 503, keep serving what you can at full speed, and drop low-priority work first. Pair that with timeouts on every downstream call, circuit breakers on dependencies that are failing, and fallbacks (stale cache, reduced response) where users can tolerate them.
7. Watch the right numbers
- Latency percentiles (p50, p95, p99), not averages.
- Error rate by status class, and 429 and 503 counts.
- Saturation: CPU, connection pool use, queue depth, thread pool use.
- Per-dependency latency, so you can see which backend is dragging the tier down.
Do
Load test to the point of failure, learn what breaks first, and set limits below it. Give every call a timeout, every list a maximum page size, and every client a rate limit.
Don't
Fix a slow API by adding instances without checking what they wait on. If the bottleneck is one database query, ten more servers just send ten times as many requests to it.
Takeaway
A scalable API layer is stateless, bounded and protective. Keep instances interchangeable, make the common path cheap with caching and cursors, push slow work to queues, enforce limits for fairness, and shed load before everything gets slow together. Then adding servers actually helps.
Frequently asked questions
Why must API servers be stateless to scale?
If any instance can answer any request, you can add or remove instances freely and a load balancer can route anywhere. State held in one server's memory, such as sessions or uploads in progress, pins users to that server and loses data when it dies. Move state to shared stores.
How should an API respond when it is overloaded?
Fail fast and visibly: return 429 for a client over its limit and 503 with Retry-After when the service is saturated. Rejecting excess work quickly protects the requests already in flight, whereas queueing everything indefinitely makes all of them slow and eventually time out.
What can an API cache safely?
Responses that are the same for many callers and tolerate brief staleness: public reads, reference data, computed aggregates. Use HTTP cache headers and ETags for clients and a shared cache like Redis for server-side results. Never cache per-user data under a shared key.
When should API work become asynchronous?
When it takes more than a second or two, depends on slow third parties, or can be retried independently: report generation, media processing, bulk imports. Accept the request, return 202 with a status URL, and do the work on a queue so request threads stay free.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.