---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 5 min
---

How Spanner Gets Global Consistency: Paxos Splits, TrueTime and Commit Wait

Splits replicated by Paxos, timestamps from bounded-error clocks, and a deliberate wait before commit. How a SQL database stays strongly consistent worldwide.

– –

The trade-off between strong consistency and global scale looks like physics until you notice the missing ingredient was a better clock. Spanner’s central move is to make time a bounded-error service and then spend a few milliseconds to be certain.

The structure

ConceptWhat it is
UniverseThe whole deployment, spanning many zones
ZoneA unit of administrative and failure isolation, with its own servers
SpanserverA server holding tablets (shards of data) and running the replication and transaction logic
SplitA contiguous range of keys, the unit of sharding and replication
Paxos groupThe replicas of one split across zones, with a leader
ColossusThe distributed file system that stores the data, decoupling storage from compute

Data is key-value with a timestamp on every version, so the database is multi-versioned. Splits grow, shrink and move automatically: a split that gets too big or too hot is divided and redistributed, so load spreads without anyone resharding by hand.

Replication: Paxos per split

Each split has replicas in several zones. One is the leader, which orders and coordinates all writes for that split. The others are followers, which can serve reads and stand ready to take over.

  • A write is committed only after a majority of the group has it. Losing a minority of replicas, even a whole zone, loses nothing.
  • If the leader fails, the group elects a new one, with no operator involvement.
  • Followers serve reads, which spreads load off the leader.

This is the same machinery that keeps a replicated log in agreement, applied per split. Since there are thousands of splits, there are thousands of independent leaders, so write load spreads across the fleet.

TrueTime: time with an error bar

Every transaction needs a timestamp that respects real-time order. Ordinary clocks cannot promise that, because they drift. TrueTime is a clock API built on GPS receivers and atomic clocks in every data centre, and servers synchronise to them frequently. The twist is the return value.

TT.now() -> [earliest, latest]     // true time is guaranteed to lie inside

Typical uncertainty is a few milliseconds. The two reference types back each other up: if GPS signals are disrupted, atomic clocks keep time, and GPS corrects the atomic clocks’ slow drift.

A write, step by step

  1. Lock

    The split’s leader takes write locks on the rows. Conflicting transactions wait. Deadlocks are avoided with wound-wait: an older transaction aborts a younger one that holds a lock it needs, and a younger one waits for an older one.

  2. Choose a timestamp

    The leader picks a commit timestamp at or above TT.now().latest, larger than any it has assigned before.

  3. Replicate to a majority

    The leader proposes the change to its Paxos group and waits for a majority to acknowledge it. Now it is durable.

  4. Commit wait

    The leader waits until its chosen timestamp is definitely in the past, meaning TT.now().earliest has passed it. Only then does it release locks and acknowledge the client.

Why waiting makes it correct

Suppose transaction A commits and the client then starts transaction B anywhere in the world. Because A waited, its timestamp is already past by the time anyone could learn it committed, so B’s timestamp is guaranteed larger. Real-time order becomes timestamp order. The price is roughly the clock uncertainty, a few milliseconds, added to commit latency.

Writes across splits. If a transaction touches rows in several splits, their leaders run two-phase commit: one leader coordinates, all participants lock and vote, and the coordinator commits only if everyone agrees. Each participant is itself a Paxos group, so a participant failing in the middle does not lose the transaction’s state.

Advertisement

Reads without locks

Because every row version has a commit timestamp, a read-only transaction picks a timestamp and reads the latest version at or before it. It needs no locks and cannot block writers.

Read typeBehaviourUse when
Strong readReturns the latest committed data. The replica confirms it is up to date, possibly asking the leaderCorrectness matters most
Stale readReturns data as of a bounded time ago (say 10 seconds) from the nearest replica, no leader round tripLatency matters and slight staleness is fine
Snapshot at timestampA consistent view of the database at an exact past timeAudits, debugging, consistent exports

What TrueTime enables

  • External consistency for all transactions.
  • Lock-free read-only transactions at a consistent snapshot.
  • Atomic schema changes, since a schema update is a transaction with a timestamp that every server honours.
  • Historical reads, for free, because versions are kept.

Costs and trade-offs

CostWhy
Commit latency includes cross-zone Paxos and commit waitStrong guarantees are paid for in milliseconds
Unavailability if a majority cannot be reachedIt is a consistency-first system: it stops writes on a split rather than diverge
Needs excellent clocks and a private networkSpecialised infrastructure, hidden by the managed service
Cross-split transactions are slowerTwo-phase commit adds round trips, so design data locality

Do

Choose it when you need relational semantics, global scale and strong consistency together. Design keys so related rows share a split, avoid hotspots from monotonically increasing keys, and use stale reads for latency-sensitive paths that tolerate them.

Don't

Use a globally consistent database for a workload that fits on one Postgres, or for data that can be eventually consistent. You pay in cost and latency for guarantees you do not use.

Takeaway

Spanner’s trick is to stop treating clock error as something to hide and instead measure it, publish it as an interval, and wait it out. Add Paxos replication per key range and multi-version storage, and you get strongly consistent, horizontally scalable SQL. The engineering lesson travels well: when you cannot remove uncertainty, bound it and make correctness depend on the bound.

Advertisement

Frequently asked questions

What is external consistency?

A guarantee stronger than serializability: if transaction A commits before transaction B starts, in real time, every observer sees A's effects before B's, and A's timestamp is lower. The database behaves as if all transactions ran one at a time in real-time order, even across continents.

What is commit wait in Spanner?

After choosing a commit timestamp, the leader waits until that timestamp is definitely in the past according to the clock's uncertainty interval before making the commit visible. It costs a few milliseconds, and it guarantees later transactions anywhere receive larger timestamps.

How does Spanner avoid locking for reads?

Data is multi-versioned with commit timestamps. A read-only transaction picks a timestamp and reads the version of each row at or before it, from any sufficiently up-to-date replica, without taking locks. Strong reads ask the replica to confirm it is current, stale reads allow bounded staleness for lower latency.

Does Spanner break the CAP theorem?

No. It chooses consistency over availability during partitions, and achieves very high availability in practice through engineering: Paxos majorities across zones, private networks and operational discipline. When a majority cannot be reached, the affected split stops accepting writes.

Sources & further reading

/* Comments */