Reducing Latency: Find the Layer First, Then Pull the Right Lever
Network, processing, database and rendering delay need different fixes. Caching, CDNs, indexes, async work, compression, connection reuse and tail latency.
On this page
Teams love to “optimise” things that were never slow. A faster algorithm on a request that spends 8 ms computing and 600 ms waiting on a distant database is theatre. The discipline is to locate the delay before touching anything.
Three words that get confused
| Term | Meaning | Analogy |
|---|---|---|
| Latency | Time for one request to complete, usually in ms | How long a delivery takes |
| Bandwidth | Maximum data a link can carry per second | How wide the road is |
| Throughput | Data actually delivered per second | Cars that really arrive per hour |
You can have huge bandwidth and terrible latency, for example a satellite link. For small, chatty requests, latency dominates. For big downloads, bandwidth does.
Why it costs money
Slow pages raise bounce rates and lower conversion. Real-time tools such as meetings and collaborative editors become unusable when delay exceeds a few hundred milliseconds. Search engines factor page experience into ranking, with Core Web Vitals including responsiveness (now Interaction to Next Paint) and load metrics. Large operators have published experiments showing that adding a few hundred milliseconds measurably reduces usage, which is why speed budgets exist.
Locate the delay
A request trace with timings per hop is the single best tool. Without one, add timing headers (Server-Timing) and browser performance marks first.
The levers, matched to the problem
| Lever | Helps with | What it costs |
|---|---|---|
| CDN | Network distance for static and cacheable content | Cache invalidation, configuration |
| Caching (browser, edge, application, database) | Repeated work and reads | Staleness, invalidation bugs |
| Connection reuse (keep-alive, HTTP/2, pooling, TLS session resumption) | Handshake round trips | Connection limits, idle timeouts to tune |
| Compression (gzip, Brotli, image formats) | Transfer time for large payloads | CPU, and useless for already-compressed data |
| Load balancing | Queueing on an overloaded server | Another hop, health checks to maintain |
| Asynchronous processing | Slow work on the request path | Eventual results, retry handling, visibility |
| Database indexing and query tuning | Scans and slow queries | Slower writes, more storage |
| Pre-caching and prefetch | Predictable next requests | Wasted bandwidth when wrong |
| Fewer round trips (batching, parallel calls) | Chatty clients and services | Larger single requests |
Every round trip costs a fixed time set by distance, whatever the payload. Merging three sequential calls into one parallel call saves more than shaving 20 percent off each body. Look at the waterfall for serial dependencies first.
Mind the tail
When a page needs data from many backends, it is as slow as the slowest of them. If each backend is slow one time in a hundred, a request that touches a hundred of them is slow most of the time. This is why percentile latency is the real metric.
Tail-trimming techniques:
- Timeouts and bounded retries with backoff and jitter, so a slow call does not hold everything.
- Hedged requests: after a short delay, send a duplicate to another replica and take the first answer, for read-only idempotent calls.
- Load shedding: reject excess work fast, instead of letting queues grow.
- Isolating slow tenants and queries so they cannot drag the rest.
A method that works
- Set a target on a percentile
For example, p95 under 300 ms for checkout. Averages are not targets.
- Measure each layer
Record DNS, connect, TLS, server time, database time and render time separately, for real users, not just your laptop.
- Fix the largest slice first
The biggest contributor to the slow percentile decides the next change.
- Change one thing, then re-measure
Otherwise you cannot tell which change helped, or which hurt.
- Guard it
Put the latency budget into monitoring and alerts so a regression is noticed before users complain.
Do
Track percentiles by endpoint, test from where your users are, and keep a written latency budget per page or API call, split by layer.
Don't
Add a cache first and ask questions later. An unmeasured cache hides the slow path, adds invalidation bugs, and often fixes nothing because the delay was somewhere else.
Takeaway
Latency work is a search problem before it is an engineering problem. Identify whether the delay is network, processing, database or rendering, apply the lever that matches, cut round trips before bytes, manage the tail with timeouts and shedding, and verify every change against a percentile target.
Frequently asked questions
What is the difference between latency, bandwidth and throughput?
Latency is the time one request takes, for data to go out and a response to come back. Bandwidth is the maximum data a link can carry per second. Throughput is the amount actually delivered per second. A fat pipe with a long round trip still feels slow for small requests.
Why measure p95 and p99 instead of the average?
Averages hide the slow requests that users actually remember. A system with a 50 ms average can have a 2 second 99th percentile. At scale one user request often fans out to many backend calls, so rare slow calls become common slow pages. Track percentiles and set targets on them.
What is the quickest win for network latency?
Put content closer to the user with a CDN and reuse connections so you skip repeated TCP and TLS handshakes. Cutting round trips beats optimising bytes, because each round trip costs a fixed time regardless of payload size, and distance sets a floor you cannot code around.
How do I decide whether to make something asynchronous?
If the user does not need the result to continue, take it off the request path: send the email, resize the image, update the analytics in the background. Return fast with an acknowledgement. Keep it synchronous only when the next step depends on the answer.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.