---
topic: system-design
author: Crashtech Editorial
published: Nov 15, 2026 · read: 5 min
---

Scale an API by Removing Contention: Stateless Workers, Safe Caches, and Bounded Queues

A practical API scaling sequence: remove instance-local state, cache only safe responses, bound work queues, and shed excess load before dependencies fail.

– –

A scalable API is not a diagram with more boxes. It is a system whose busiest dependency remains inside its capacity envelope as traffic shifts, cache misses cluster, and downstream services slow down.

Start with the bottleneck

Before changing architecture, measure request volume, successful throughput, latency percentiles, error rate, and saturation at each dependency. A rising request count with stable CPU but a growing database connection wait is a different problem from a CPU-bound serializer. Tail latency often reveals pressure that an average conceals.

Define a capacity unit in product terms: completed checkouts per second, documents indexed per minute, or reads served under a latency objective. Then estimate the limiting resource and reproduce the load shape, including bursts and skewed keys. A benchmark that spreads requests evenly can miss a single tenant, hot item, or slow query that dominates production.

Horizontal scaling is useful when workers are interchangeable and the next dependency has room. If every new worker opens a large connection pool, scaling the API tier may overwhelm the database sooner. Set per-instance concurrency and connection limits from the downstream budget, not from the number of CPU cores alone.

Make request handling shareable

A worker is operationally stateless when any healthy instance can continue a request without relying on hidden memory held by the instance that received the previous request. User sessions, job state, and idempotency records need explicit stores or tokens with defined lifetime and revocation behavior.

This does not remove state from the system; it moves it to a component with a deliberate contract. A session store can become a bottleneck. A signed token can be difficult to revoke immediately. A durable job queue can duplicate delivery. Make those tradeoffs visible before using statelessness as a synonym for easy scale.

Keep request work bounded: validate inputs early, cap body size, use caller deadlines, avoid unbounded fan-out, and stop waiting when a dependency cannot return useful work in time. For operations that may be retried after a client timeout, an idempotency key can let the server distinguish a safe retry from a second side effect. AWS’s guidance emphasizes that the token must be tied to the operation’s parameters and retained for an appropriate period; a key with ambiguous reuse semantics can be dangerous. [4]

Cache with an identity and freshness contract

Caching reduces repeated work, but a cache is correct only when its key identifies the response and its freshness rule matches the product. NGINX’s proxy cache documentation exposes these mechanics directly: a cache key can include request properties, response headers affect caching, and proxy_cache_lock can limit how many concurrent misses populate the same entry. [1]

Before caching an endpoint, answer four questions:

  1. Who may see the response? A cache key must not collapse two authorization scopes into one entry.
  2. What makes the response change? Choose a TTL, version, or invalidation event that reflects the source of truth.
  3. What can be stale? Search suggestions may tolerate a short delay; account balances may not.
  4. What happens on a miss or purge? Cold caches can create a thundering herd or send a burst to the origin.

Cache public, stable, read-heavy representations first. Treat personalized content conservatively. Test behavior for authorization headers, cookies, Vary, errors, and stale responses. A cache hit ratio by itself is not a success metric if the system is serving the wrong user’s data or hiding an update users expect to see.

Use queues for work that can finish later

A queue separates request acceptance from background execution. It helps when a task is slow, bursty, and can be completed asynchronously. It does not make the work disappear. If the producer continuously outpaces consumers, queue age and storage use grow until users wait longer or the system runs out of room.

Set explicit limits for queue depth, message size, retention, worker concurrency, and retry attempts. Decide whether the API rejects, defers, or degrades when admission is closed. Expose job status with an identifier and document the expected completion behavior. Make handlers idempotent because redelivery is normal in many queue systems.

For synchronous requests, use bounded queues or no queue at all. Google’s SRE guidance explains how queues can increase latency and memory use during overload, while retries can amplify demand and feed cascading failures. Load shedding protects useful work by rejecting some requests before every worker is saturated. [2] [3]

Protect the dependency that sets your limit

Rate limits, concurrency limits, and timeouts solve different problems. A rate limit controls arrivals over time; a concurrency limit caps work in flight; a timeout bounds how long a caller waits. Apply limits at the boundary where you can identify the tenant or workload, and protect shared dependencies with their own budgets.

Retries need special care. A caller that retries every error immediately can double the load during an outage. Use a bounded retry policy with backoff and jitter, honor the original deadline, and retry only operations whose semantics allow it. Avoid retrying at multiple layers without a shared attempt budget.

Observe queue age, cache miss bursts, in-flight work, connection-pool wait, rejection rates, retry volume, and end-to-end success. Alert on customer outcomes and saturation, not just instance health. For further reading, see token-bucket rate limiting and uncertain outcomes in distributed calls.

A sequence for the next scaling review

  1. Find the saturated resource and state the desired operation-level capacity.
  2. Remove hidden per-instance state, but document the new state store’s limits.
  3. Add caching only where authorization, freshness, and invalidation are testable.
  4. Move only asynchronous-capable work to a bounded, durable queue.
  5. Add concurrency limits, deadlines, load shedding, and idempotent retry behavior.
  6. Re-run a burst and failure test with hot keys and a slow downstream dependency.

A production API scales when the system controls the amount of work it admits and protects its limiting dependency. More servers can help, but only after the request path has a safe state model, a bounded workload, and an honest overload response.

Advertisement

Frequently asked questions

Does stateless mean an API has no state?

No. It means an individual application instance does not hold the only copy of session or workflow state that another instance needs. State can live in a database, cache, or durable queue, each with its own consistency and failure tradeoffs.

Should every API response be cached?

No. Cache only responses with a clear identity, authorization, freshness, and invalidation contract. Personalized or rapidly changing responses can leak data or become incorrect when the cache key or purge path is incomplete.

Why can a queue make an overloaded API worse?

An unbounded queue converts immediate rejection into delayed resource exhaustion. Bound queue length and age, define admission and shedding behavior, and size consumers against the downstream service rather than just the producer rate.

Sources & further reading

/* Comments */