How should an API respond when it is overloaded?
Fail fast and visibly: return 429 for a client over its limit and 503 with Retry-After when the service is saturated. Rejecting excess work quickly protects the requests already in flight, whereas queueing everything indefinitely makes all of them slow and eventually time out.
Answered in
Scaling the API Layer: Stateless Servers, Rate Limits, Caching and Async WorkThe API tier is the easy part to scale, if you keep it stateless. The practices that matter: limits, pagination, caching, queues and graceful overload.
Read the full analysisOther questions this article answers
More system design & architecture questions
- What is the architectural difference between an API Gateway and a Reverse Proxy?
- When should an architecture use Layer 4 instead of Layer 7 load balancing?
- How do edge ingress layers prevent backend cascading thundering herds?
- What algorithm is best for sticky sessions without breaking even distribution?
- How can a CDN improve performance for dynamic requests?
- What is the hardest part of operating behind a CDN?
- How can a CDN help with traffic surges or DDoS attacks?
- What is the difference between a Push CDN and a Pull CDN?
Every answer on Crashtech is written by the editor of the article it comes from — never auto-summarised. Browse all answers or the System Design & Architecture beat.