---
topic: distributed-systems
author: Crashtech Editorial
published: Jul 23, 2026 · read: 3 min
---

Whose Clock Is Right? Ordering Events Across Machines Without a Shared Time Source

Why physical clocks drift in distributed systems, and how Lamport timestamps, vector clocks, and TrueTime establish causal order.

– –

Every software engineer learns early on to timestamp events. When debugging a single process running on a laptop, ordering is simple: take the system clock, record System.currentTimeMillis(), and sort the logs. The event with the smaller number happened first.

In a distributed system, this assumption is false. There is no such thing as “now” across a fleet of independent servers. Each machine possesses its own physical quartz oscillator, and quartz crystals drift based on ambient server room temperature, electrical frequency variations, and hardware aging.

Relying on wall-clock timestamps to determine the order of distributed writes leads directly to silent data loss, split-brain conditions, and broken consistency guarantees. Understanding how distributed systems establish causality without a universal clock is one of the foundational requirements of systems architecture.

Why Physical Clocks Drift and Fail

Computers track time using quartz crystal clocks that count microscopic vibrations. Even high-grade oscillators drift by a few parts per million, which translates to a drift of several milliseconds every single day.

Server A (Cool Rack):    12:00:00.000 ms
Server B (Warm Rack):    12:00:00.042 ms  (+42ms Clock Skew)
Server C (CPU Throttled): 11:59:59.980 ms  (-20ms Clock Skew)

Engineers typically attempt to solve this with the Network Time Protocol (NTP). NTP servers periodically exchange UDP packets to measure network round-trip time and calculate clock offsets. However, NTP has major limitations:

  • Variable Network Asymmetry: If the outbound network path takes 15 ms and the return path takes 45 ms, NTP miscalculates the offset by 15 ms.
  • Clock Slew and Steps: If a clock drifts too far, NTP either steps backward (which breaks monotonic counters) or slews time forward gradually, leaving significant uncertainty windows.
  • Virtualization Pauses: In cloud environments like AWS or GCP, a hypervisor “stop-the-world” pause during VM live migration can suspend a virtual machine for hundreds of milliseconds, instantly creating severe clock skew.

If two database nodes use local timestamps to implement Last-Write-Wins (LWW), a write that occurred later in real physical time can be overwritten and silently erased by an earlier write whose node clock was skewed ahead.

Lamport Timestamps: Ordering by Logical Causality

In 1978, Leslie Lamport published a breakthrough insight: distributed systems rarely need to know the exact physical time an event occurred. They only need to know whether Event A caused Event B. This is the happens-before relationship ($A \rightarrow B$).

Lamport introduced Logical Clocks, where every node maintains a simple integer counter:

  1. Before executing an internal event, Node $i$ increments its counter: $C_i = C_i + 1$.
  2. When sending a network message, Node $i$ attaches its current counter: $(msg, C_i)$.
  3. When Node j receives (msg, C_i), it updates its own clock to be strictly greater than the sender’s clock before processing:
C_j = max(C_j, C_i) + 1
Node 1:  [E1: C=1] ---> (Send msg, C=1) 
                            \
                             \
Node 2:  [E1: C=1]           \---> [Recv msg: max(1, 1)+1 = 2] -> [E3: C=3]

Lamport clocks guarantee that if event A -> B, then C(A) < C(B).

However, the converse is not true. If C(A) < C(B), you cannot conclude that A caused B; the two events might have occurred concurrently on separate nodes with no communication between them.

Vector Clocks: Detecting True Concurrency

To distinguish causal dependency from concurrent events, distributed systems use Vector Clocks.

Instead of a single integer, every node maintains an array (vector) of size $N$, where $N$ is the number of participating nodes in the cluster. Position $k$ represents Node $k$‘s latest known sequence number.

  1. Initially, all nodes start with $[0, 0, \dots, 0]$.
  2. Node $i$ increments its own index $V_i[i]$ before any local event.
  3. Node $i$ attaches its entire vector $V_i$ to every outgoing message.
  4. When Node j receives vector V_msg, it merges the vector by taking the element-wise maximum across all positions and then increments its own index:
V_j[k] = max(V_j[k], V_msg[k])  for all k
V_j[j] = V_j[j] + 1

Comparing Two Vector Timestamps

Given two events with vector timestamps $V_A$ and $V_B$:

  • Causally Ordered: V_A < V_B if every element in V_A <= V_B and at least one element is strictly smaller. This proves Event A happened before Event B.
  • Concurrent (Conflict): If V_A[1] > V_B[1] but V_A[2] < V_B[2], neither caused the other. The events occurred concurrently.

Amazon Dynamo and Riak famously adopted vector clocks to detect concurrent shopping cart updates during network partitions, presenting both branches to the client application for conflict resolution.

       Vector A: [3, 1]              Vector B: [2, 4]
       (Node 1 has seen more)        (Node 2 has seen more)
       
              Result: CONFLICT (Concurrent Branching)
              Neither dominates. Requires merge resolution.

The primary operational drawback of vector clocks is state bloat: in a cluster with thousands of dynamic nodes or microservices, vector sizes grow proportionally to the participant count unless aggressive pruning heuristics are applied.

Google TrueTime: Taming Physical Clocks with Hardware

In 2012, Google published the design of Spanner, a globally distributed database that provides external consistency (serializability) using physical time. Rather than relying on purely logical counters or accepting NTP’s unbounded drift, Google engineered TrueTime.

TrueTime equips every Google data center with redundant atomic clocks (rubidium oscillators) and GPS antenna receivers:

  • GPS clocks are highly accurate but can fail due to antenna issues or signal degradation.
  • Atomic clocks do not rely on satellite signals and drift at predictable, minute rates.

Because physical drift cannot be eliminated entirely, TrueTime does not return a single point in time. Instead, the TrueTime.now() API returns an explicit uncertainty interval:

[t.earliest, t.latest]  where  (t.latest - t.earliest) = 2 * epsilon

In Google’s infrastructure, the error bound $\epsilon$ is guaranteed to remain under 7 milliseconds (and typically under 1 ms).

+-------------------------------------------------------------+
|                     TRUETIME WAIT RULE                      |
|                                                             |
| Node commits Tx1 at [t_1 - eps, t_1 + eps].                 |
| Commit Timestamp s = t_1 + eps                              |
|                                                             |
| Wait: Thread sleeps until TrueTime.now().earliest > s       |
|                                                             |
| Guarantee: Any subsequent Tx2 receives s2 strictly > s.     |
| Physical time now guarantees absolute serializability.     |
+-------------------------------------------------------------+

To guarantee that a transaction $T_2$ initiated after $T_1$ receives a strictly higher timestamp, Spanner implements the Commit Wait Rule: the database engine deliberately pauses execution for $2\epsilon$ before releasing transaction locks. By waiting out the clock uncertainty window, Spanner ensures that no overlapping time intervals can exist between causally ordered transactions.

Choosing the Right Ordering Primitive

MechanismClock SourceStorage OverheadCausality GuaranteeTradeoff
Physical LWWLocal quartz + NTPNone ($8$ bytes)None (Skews cause silent data loss)High risk of data loss under network partitions
Lamport ClockLogical integer8 bytes per nodePartial (A -> B => C_A < C_B)Cannot distinguish concurrency from causality
Vector ClockLogical vector$O(N)$ vector arrayFull (Identifies exact concurrency)Array size bloats as node count expands
TrueTimeGPS + Atomic ClocksNoneFull global serializabilityDemands proprietary data center hardware

Ordering events in a distributed system is fundamentally about choosing which uncertainty you are willing to accept: unbounded logical vector sizes, application-level merge logic, or hardware-enforced commit-wait pauses.

Advertisement

/* Comments */