---
topic: system-design
author: Crashtech Editorial
published: Nov 17, 2026 · read: 5 min
---

Circuit Breakers, Bulkheads, Timeouts: The Patterns Behind 'Fault Tolerant'

Fault tolerance isn't a single feature you add — it's four specific patterns, each stopping a different way one failure turns into every failure.

– –

Distributed systems fail constantly — a node crashes, a network link degrades, a downstream dependency times out — and none of that is really the design problem. The design problem is what happens next: does one failing component quietly take the rest of the system down with it, or does the system absorb the failure and keep functioning for everything that failure didn’t directly touch. Four specific, well-understood patterns are what separate those two outcomes, and each one is answering a different question about how failure spreads.

Timeouts: bounding how long you’ll wait, not preventing the wait

The most basic pattern, and the one most systems already have in some form: cap how long a caller will wait for a response before giving up. Without one, a single slow or hung dependency can leave a caller waiting indefinitely, tying up a thread, a connection, or a request-handling slot for as long as the dependency stays broken.

What a timeout alone doesn’t solve: if the dependency stays down, every new request still has to wait out the full timeout duration before failing — slow, repeated failure, one request at a time, rather than fast failure. Under real load, that’s enough hung, timeout-bound requests to exhaust the caller’s own capacity even though the caller’s own code is working fine.

Circuit breakers: stop calling something that’s clearly broken

A circuit breaker tracks recent failure rate against a threshold and, once crossed, stops attempting calls to that dependency entirely for a cooldown period — failing immediately instead of waiting out a timeout on every single request.

CLOSED (normal)  → failures exceed threshold →  OPEN (fail fast, no calls attempted)
                                                        │
                                              cooldown period elapses
                                                        ▼
                                          HALF-OPEN (allow a few test requests)
                                            │                        │
                                      test succeeds              test fails
                                            ▼                        ▼
                                        CLOSED                    OPEN again

This protects two things at once: the caller stops wasting resources waiting on calls that are very likely to fail anyway, and the struggling dependency stops receiving additional load exactly when it’s least able to handle it — which is often precisely what it needs to actually recover.

A circuit breaker changes the failure mode, not the failure

It doesn’t fix the underlying dependency — it converts “every caller waits out a timeout on every request” into “callers fail fast and the dependency gets breathing room to recover.” That’s a meaningfully better failure mode even though the dependency is still down.

Advertisement

Bulkheads: isolating resources so one failure can’t starve everything else

Named for the watertight compartments in a ship’s hull — a breach in one compartment floods that compartment, not the whole ship. Applied to software: give each dependency its own isolated pool of resources (threads, connections) instead of sharing one pool across every downstream call a service makes.

Without bulkheads shared pool, shared risk
One slow or failing dependency consumes threads or connections from a shared pool until none are left — starving calls to completely unrelated, perfectly healthy dependencies that happen to draw from the same pool.
With bulkheads isolated pools, contained damage
Each dependency gets its own dedicated pool. A failing dependency can exhaust its own allocation, but calls to every other dependency continue drawing from their own, unaffected pools.

This is the pattern circuit breakers alone don’t cover: a circuit breaker responds to one dependency being unhealthy by stopping calls to that dependency. A bulkhead prevents that dependency’s resource consumption from ever being able to affect calls to a different dependency in the first place — a structural containment measure, not a reactive one.

Backoff with jitter: don’t let recovery become a second outage

When a failed request is retried, doing so immediately — especially from many clients simultaneously — risks a specific failure mode: the retries themselves overwhelm a service that was just starting to recover, turning a brief blip into a sustained outage triggered by the recovery attempt itself.

  1. Exponential backoff

    Space retry attempts out with increasing delay — 1s, then 2s, then 4s, then 8s — instead of retrying immediately or at a fixed interval, giving a struggling dependency actual time between attempts to stabilize.

  2. Add jitter

    Randomize the delay within each backoff window. Without jitter, many clients that failed at the same moment retry in synchronized waves, each wave itself a mini traffic spike. Jitter spreads those retries out in time, smoothing the resulting load instead of concentrating it.

Do

Combine backoff and jitter together — backoff alone still allows synchronized retry waves; jitter alone still allows retries to come too fast on average. Together, they space retries out both across time and across clients.

Don't

Retry immediately and repeatedly on failure, especially from many clients at once. This is precisely the mechanism that turns a transient blip into a sustained, self-inflicted outage right as a service is trying to come back.

Why all four, and not just the one that seems most relevant

Timeout alone gap: slow failure at scale
Bounds individual wait time, but every request still pays that wait during an outage, which can exhaust caller capacity under load.
Circuit breaker alone gap: shared resource exhaustion
Stops calling one failing dependency, but doesn’t prevent that dependency’s resource usage from starving calls to unrelated, healthy dependencies before the breaker trips.
Bulkhead alone gap: no fail-fast, no recovery protection
Isolates resource pools, but without a circuit breaker, a caller still waits out full timeouts on a failing dependency’s isolated pool, and without backoff, retries can still storm a recovering service.
Backoff/jitter alone gap: no fast-fail, no isolation
Prevents retry storms, but without a circuit breaker, individual requests still attempt (and wait to fail against) a known-broken dependency each time.

Each pattern closes a specific gap the others leave open — which is why production resilience engineering treats them as a composed set, not a menu to pick one item from.

Takeaway

“Fault tolerant” describes an outcome, not a single mechanism, and the four patterns behind it are each answering a distinct question: how long do we wait (timeouts), when do we stop trying (circuit breakers), how do we contain the damage (bulkheads), and how do we avoid re-breaking something that’s recovering (backoff with jitter). Distributed systems will keep failing in ways no amount of design prevents outright — nodes crash, networks partition, dependencies time out. What these four patterns actually buy you is containment: the failure that was always going to happen stays scoped to what it directly touched, instead of cascading into everything downstream of it.

Advertisement

Frequently asked questions

Why isn't a timeout by itself enough to make a system fault tolerant?

A timeout stops one caller from waiting forever on one slow request — but if a downstream dependency stays broken, every subsequent call still has to wait out the full timeout before failing, one request at a time. That's slow failure, not fast failure, and under load it can tie up so many resources waiting on doomed requests that the caller itself becomes unresponsive. Timeouts are necessary but not sufficient on their own.

How does a circuit breaker actually decide when to stop calling a failing dependency?

It tracks recent failures against a threshold. Once failures cross that threshold, the breaker 'opens' — for a cooldown period, calls fail immediately without even attempting the dependency, protecting both the caller (no more wasted waiting) and the struggling dependency (no more added load while it's already failing). After the cooldown, the breaker allows a small number of test requests through; if those succeed, it closes again and normal calls resume.

What's the difference between a bulkhead and just having a circuit breaker?

A circuit breaker protects a caller from one specific failing dependency by stopping calls to it. A bulkhead protects a caller's own resources — thread pools, connection pools — from being exhausted by any single dependency in the first place, by giving each dependency its own isolated pool. They solve related but distinct problems: a circuit breaker responds to a dependency being unhealthy; a bulkhead prevents one dependency's resource consumption from starving calls to a completely different, healthy dependency.

Why does retrying a failed request need exponential backoff instead of just retrying immediately?

Retrying immediately, especially from many clients at once, can turn a transient blip into a sustained overload — every client hammering the same struggling service at the same instant, right as it's trying to recover. Exponential backoff spaces retries out with increasing delay between attempts, and adding random jitter to that delay prevents many clients from retrying in synchronized waves, both of which give a recovering system actual room to stabilize.

Can these four patterns be applied independently, or do they need to work together?

They compose, and are strongest together: a timeout bounds how long any single call waits, a circuit breaker stops calling a dependency that's clearly unhealthy, a bulkhead ensures that dependency's failure can't exhaust resources needed for calls to other services, and backoff-with-jitter prevents retries from turning a recovering dependency's comeback into a second outage. Each one closes a specific gap the others leave open — using only one or two still leaves a real failure mode unaddressed.

/* Comments */