Distributed Systems Fundamentals: Partial Failure, No Shared Clock, and What You Do About Them

On one machine, a function call either returns or the program dies. Between two machines there is a third outcome: you called, and you will never know what happened. Almost everything interesting in distributed systems is a response to that single fact.

What changes when you leave one machine

Search, online banking and multiplayer games all work this way. The machinery is invisible until it breaks.

The three questions behind most bugs

1. Did it happen?

You send a payment request and the connection times out. Three worlds are indistinguishable from your side:

World A: the request never reached the server       -> safe to retry
World B: the server processed it, the reply was lost -> retry double-charges
World C: the server crashed halfway                  -> state may be partial

You cannot tell them apart from the client. The answer is not to guess better but to make retry safe: give each operation an idempotency key, so repeating it has the same effect as doing it once.

2. In what order?

Two nodes each record an event with their own clock. Their timestamps disagree by milliseconds, which is plenty to reverse cause and effect. Lamport’s classic paper showed that what matters is the happened-before relation, and that a logical counter attached to messages can order causally related events without any synchronised clock.

Node A:  e1 (counter 1) ---- message ----> 
Node B:                                   e2 (counter max(0,1)+1 = 2)

When you need a single agreed order of writes, you use a consensus protocol such as Raft or Paxos instead of trusting clocks.

3. Who is in charge?

Replicas must agree on a leader, or two nodes both think they are primary and accept conflicting writes. Leader election, quorums and fencing tokens exist to answer this safely, even when the network splits.

The fallacies that bite

The “fallacies of distributed computing” are assumptions that hold on a developer laptop and fail in production.

The standard toolkit

1. Timeouts and bounded retries

   Fail fast, retry a small number of times, and back off exponentially with jitter so a recovering service is not stampeded by synchronised retries.

2. Idempotency

   Design writes so repeating them is safe: idempotency keys, upserts, conditional updates. This is what makes retries legal.

3. Replication

   Keep copies on several nodes so one failure does not lose data or availability. You then choose how strongly replicas must agree, which is the consistency trade-off.

4. Partitioning

   Split data across nodes so no single machine must hold or serve everything.

5. Consensus and leader election

   Use a proven protocol for decisions that must be unanimous, such as who the leader is or the order of writes. Do not invent your own.

6. Observability

   Distributed tracing, correlated logs and good metrics, because the failing component is rarely the one showing the error.

Benefits and costs, plainly

Takeaway

Distributed systems are hard because machines share only a network, so failure is partial, time is unreliable and certainty is expensive. Assume every call can time out, make operations safe to repeat, order events logically, and agree on leaders with proven protocols. Do that, and most of the scary vocabulary turns into a short checklist.