Distributed Systems Fundamentals: Partial Failure, RPC Ambiguity, and Clock Limits
A practical guide to uncertain RPC outcomes, deadlines, safe retries, idempotency, and why wall clocks cannot establish causality.
On this page
A payment request reaches a service, the database commits it, and then the response packet disappears. The caller sees a timeout. Should it try again? The answer depends on the operation’s contract, because the client cannot infer from silence whether the server never ran, is still working, or finished and lost the reply. That gap between what happened remotely and what the caller can observe is one of the defining problems of distributed software.

A timeout is evidence about waiting, not about execution
In a single process, a function call usually returns a value or throws an exception within the process that made it. A remote procedure call crosses a network and involves at least two independent processes. The client can stop waiting while the server continues. The server can finish and send a response that arrives after the client’s deadline. The connection itself can fail while the server’s state remains healthy. A timeout collapses these different states into one local observation: no acceptable response arrived in time.
The gRPC documentation makes the operational consequence explicit: applications should set realistic deadlines, and servers should stop work when a call is canceled where possible. When a server handles one RPC by calling another service, it should propagate the remaining budget rather than give each downstream call a fresh full timeout. Otherwise a request with a one-second user-facing budget can fan out into several seconds of abandoned backend work. Deadline propagation and cancellation behavior depend on language implementation and application code, so they must be verified in the actual stack. [3] [4]
This is why “retry on timeout” is incomplete advice. A read-only lookup may be safe to repeat. A request that charges a card or creates a shipment may not be. The caller needs an operation-level answer to a different question: if the first attempt did succeed, can another attempt return the same result without creating another side effect?
Design retries around an operation contract
Retries can recover from short-lived faults, but they also add traffic when a service is already overloaded. Google’s SRE guidance describes how missed RPC deadlines, queued work, retries, resource exhaustion and unhealthy replicas can reinforce one another until a local failure spreads. Backoff and jitter help prevent many clients from synchronizing their retries, but they are not substitutes for limits, load shedding, or a safe operation contract. [2]
For a side-effecting request, an idempotency key can identify the logical operation across attempts. The server records the key together with the request identity and outcome. If the same key arrives again with the same operation, the service can return the stored result instead of repeating the side effect. A production design must also decide how long to retain keys, how to handle a key reused with different input, how concurrent duplicate requests are serialized, and what the client can inspect when the original result is still pending.
AWS describes idempotent API behavior as a way to make automated retries compatible with “at most once” business outcomes, while placing additional responsibility on the service implementation. That is a useful distinction: the network may deliver a request more than once, but a carefully designed API can make repeated attempts converge on one logical outcome. It is not a promise that the transport delivered the packet once. [5]
For example, a checkout service can assign one operation key when the user confirms the purchase. If the browser retries after losing the response, the server checks the key and returns the previous order result. A new user action gets a new key. The server should reject a duplicate key whose material request fields differ, because silently treating two different purchases as the same operation can conceal a client bug or abuse.
Use clocks for timeouts, not universal event ordering
Physical clocks are necessary for schedules, expiry and human-readable timestamps, but they are imperfect: clocks drift, machines synchronize at different times, and timestamp precision does not prove causality. Lamport’s classic paper formalizes “happened before” as a partial ordering. If one process sends a message and another receives it, those events are causally ordered. Two independent events with no message path between them may be concurrent, even if their displayed timestamps suggest a winner. [1]
Logical clocks can assign values consistent with causal order, but a total ordering built from logical timestamps is still a chosen ordering, not proof that two real-world events were physically simultaneous or that one caused the other. Systems that need stronger guarantees—such as unique ownership, serializable transactions or consensus across replicas—must use the corresponding coordination protocol. Replacing a logical ordering with wall-clock sorting can produce a clean-looking log while hiding concurrent writes.
Clock uncertainty also matters for deadlines. gRPC avoids sending an absolute deadline to another machine as if both clocks were perfectly synchronized; implementations can convert it into a timeout after accounting for time already spent. This leaves each hop with a remaining budget and reduces sensitivity to skew. A timeout still does not cancel work automatically in every application; handlers must observe cancellation and stop expensive downstream tasks. [3]
Trace the request across the boundary
A reliable request path usually needs three linked controls. First, a propagated deadline bounds how long each service should spend. Second, a stable operation identifier and idempotency policy define what a retry means. Third, tracing and structured logs connect the client attempt, server execution, database transaction and response. These controls answer different questions: how long to wait, whether work may be repeated, and where the request went.
When an incident occurs, engineers should distinguish “the caller timed out” from “the operation failed.” Useful records include the operation key, trace ID, deadline remaining at each hop, cancellation time, commit status and downstream response. Avoid logging credentials or full sensitive request bodies just to establish correlation. Keep metrics that reveal saturated queues and late work; Google SRE notes that averages can hide the tail behavior associated with missed deadlines and cascading overload. [2]
The Google Zanzibar architecture shows another consequence of coordinating distributed state: a system must define what ordering and consistency its authorization decisions require. For caching and replicated lookup paths, the consistent hashing guide explains how placement changes can affect which nodes receive work. These are different problems, but both become clearer when an application names its consistency contract instead of assuming the network will provide one.
The practical rule
Treat remote calls as uncertain until the application contract resolves the uncertainty. Set deadlines, propagate cancellation, make retries bounded and jittered, and give important side effects an idempotency strategy. Use causal or consensus mechanisms when the product truly requires an ordering guarantee. A distributed system cannot eliminate partial failure; it can make the ambiguity visible, bounded and safe to recover from.
Frequently asked questions
Does an RPC timeout prove that the server did not perform the operation?
No. The server may have completed the operation while its response was delayed or lost, so the client can time out without knowing whether the side effect happened. The operation contract must define safe retry behavior.
What is the difference between a timeout and an RPC deadline?
A timeout is usually a duration a caller is willing to wait; a deadline is the point in time by which the call should finish. RPC libraries may convert a deadline into a remaining timeout when forwarding work to another service.
Can synchronized wall clocks establish which distributed event happened first?
Not reliably. Physical clocks can drift and two independent events may have no causal order. Logical clocks can preserve a happened-before relation, while application-specific versions or consensus may be needed for stronger ordering guarantees.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.