---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 4 min
---

Scaling the API Layer: Stateless Servers, Rate Limits, Caching and Async Work

The API tier is the easy part to scale, if you keep it stateless. The practices that matter: limits, pagination, caching, queues and graceful overload.

– –

Adding API servers is the easy part of scaling. The hard part is making sure the extra servers do not all queue behind the same slow database, and that one enthusiastic client cannot starve everyone else.

What the layer is responsible for

Process requests against a contract, authenticate and authorise callers, orchestrate calls to backend services and aggregate results, then format and return responses. Every one of those steps is a place for latency to accumulate, and every request exercises all of them.

1. Stay stateless

Any instance must be able to serve any request. That means:

  • Sessions in a shared store or in signed tokens, not in instance memory.
  • Uploads streamed to object storage, not held on local disk.
  • Configuration from the environment, not from files edited on one box.

Then scaling is mechanical: add instances, let the load balancer spread them, remove them when traffic falls.

2. Make the cheap path cheap

Authenticate fast every request pays
Verify signed tokens locally rather than calling an auth service per request. Cache permission lookups briefly.
Bound everything size, time, fan-out
Cap request body size, page size, query depth and time per request. Unbounded inputs are how one call becomes an outage.
Paginate by cursor no deep offsets
Offset pagination gets slower the deeper you go. Cursor or keyset pagination stays fast and is stable while data changes.
Avoid N+1 calls batch downstream
One request that triggers a hundred backend calls scales as badly as the slowest of them. Batch, or load related data in a single query.

3. Cache what repeats

LayerWhatNotes
Client and CDNPublic, repeated GETs with Cache-Control and ETagFree scale. Revalidate with conditional requests
Shared cache (Redis)Computed results, lookups, rate-limit countersUse cache-aside: read cache, on miss load and store
In-processTiny, hot, rarely changing dataPer-instance copies can disagree briefly

Set an expiry on everything, decide how a write invalidates the entry, and never cache user-specific data under a key that is not user-specific.

Stampede protection

When a popular cache entry expires, hundreds of requests can miss at once and hit the database together. Let one request refresh while others serve the slightly stale value, and add random jitter to expiry times so entries do not all expire together.

4. Move slow work off the request path

If something takes seconds, depends on a flaky third party, or can be retried, do not make the caller wait on a thread.

POST /reports          -> 202 Accepted, Location: /reports/42
                          (job placed on a queue)
GET  /reports/42       -> { "status": "running" } ... { "status": "done", "url": ... }

A queue also levels load: spikes pile up in the queue and workers drain it at a steady rate, so the database behind them sees smooth demand. Make jobs idempotent, because queues deliver at least once.

Advertisement

5. Rate limit for fairness and survival

Limits stop one client from consuming everyone’s capacity and protect you from runaway scripts and retry storms. Common shapes:

LimiterProtects against
Per-client request rate (token bucket)A single caller flooding the API
Concurrency limitToo many slow requests in flight at once
Load shedderTotal demand beyond what the fleet can serve
Priority tiersLetting critical traffic through while dropping the optional

Return 429 with Retry-After and tell clients how to back off. The limiter itself needs shared state, usually a fast store, so all instances enforce the same limit.

6. Degrade, do not collapse

When demand exceeds capacity, queueing everything makes every request slow, then every request times out. Instead shed load: reject the excess quickly with 503, keep serving what you can at full speed, and drop low-priority work first. Pair that with timeouts on every downstream call, circuit breakers on dependencies that are failing, and fallbacks (stale cache, reduced response) where users can tolerate them.

7. Watch the right numbers

  • Latency percentiles (p50, p95, p99), not averages.
  • Error rate by status class, and 429 and 503 counts.
  • Saturation: CPU, connection pool use, queue depth, thread pool use.
  • Per-dependency latency, so you can see which backend is dragging the tier down.

Do

Load test to the point of failure, learn what breaks first, and set limits below it. Give every call a timeout, every list a maximum page size, and every client a rate limit.

Don't

Fix a slow API by adding instances without checking what they wait on. If the bottleneck is one database query, ten more servers just send ten times as many requests to it.

Takeaway

A scalable API layer is stateless, bounded and protective. Keep instances interchangeable, make the common path cheap with caching and cursors, push slow work to queues, enforce limits for fairness, and shed load before everything gets slow together. Then adding servers actually helps.

Advertisement

Frequently asked questions

Why must API servers be stateless to scale?

If any instance can answer any request, you can add or remove instances freely and a load balancer can route anywhere. State held in one server's memory, such as sessions or uploads in progress, pins users to that server and loses data when it dies. Move state to shared stores.

How should an API respond when it is overloaded?

Fail fast and visibly: return 429 for a client over its limit and 503 with Retry-After when the service is saturated. Rejecting excess work quickly protects the requests already in flight, whereas queueing everything indefinitely makes all of them slow and eventually time out.

What can an API cache safely?

Responses that are the same for many callers and tolerate brief staleness: public reads, reference data, computed aggregates. Use HTTP cache headers and ETags for clients and a shared cache like Redis for server-side results. Never cache per-user data under a shared key.

When should API work become asynchronous?

When it takes more than a second or two, depends on slow third parties, or can be retried independently: report generation, media processing, bulk imports. Accept the request, return 202 with a status URL, and do the work on a queue so request threads stay free.

Sources & further reading

/* Comments */