How Spanner Gets Global Consistency: Paxos Splits, TrueTime and Commit Wait
Splits replicated by Paxos, timestamps from bounded-error clocks, and a deliberate wait before commit. How a SQL database stays strongly consistent worldwide.
On this page
The trade-off between strong consistency and global scale looks like physics until you notice the missing ingredient was a better clock. Spanner’s central move is to make time a bounded-error service and then spend a few milliseconds to be certain.
The structure
| Concept | What it is |
|---|---|
| Universe | The whole deployment, spanning many zones |
| Zone | A unit of administrative and failure isolation, with its own servers |
| Spanserver | A server holding tablets (shards of data) and running the replication and transaction logic |
| Split | A contiguous range of keys, the unit of sharding and replication |
| Paxos group | The replicas of one split across zones, with a leader |
| Colossus | The distributed file system that stores the data, decoupling storage from compute |
Data is key-value with a timestamp on every version, so the database is multi-versioned. Splits grow, shrink and move automatically: a split that gets too big or too hot is divided and redistributed, so load spreads without anyone resharding by hand.
Replication: Paxos per split
Each split has replicas in several zones. One is the leader, which orders and coordinates all writes for that split. The others are followers, which can serve reads and stand ready to take over.
- A write is committed only after a majority of the group has it. Losing a minority of replicas, even a whole zone, loses nothing.
- If the leader fails, the group elects a new one, with no operator involvement.
- Followers serve reads, which spreads load off the leader.
This is the same machinery that keeps a replicated log in agreement, applied per split. Since there are thousands of splits, there are thousands of independent leaders, so write load spreads across the fleet.
TrueTime: time with an error bar
Every transaction needs a timestamp that respects real-time order. Ordinary clocks cannot promise that, because they drift. TrueTime is a clock API built on GPS receivers and atomic clocks in every data centre, and servers synchronise to them frequently. The twist is the return value.
TT.now() -> [earliest, latest] // true time is guaranteed to lie inside
Typical uncertainty is a few milliseconds. The two reference types back each other up: if GPS signals are disrupted, atomic clocks keep time, and GPS corrects the atomic clocks’ slow drift.
A write, step by step
- Lock
The split’s leader takes write locks on the rows. Conflicting transactions wait. Deadlocks are avoided with wound-wait: an older transaction aborts a younger one that holds a lock it needs, and a younger one waits for an older one.
- Choose a timestamp
The leader picks a commit timestamp at or above
TT.now().latest, larger than any it has assigned before. - Replicate to a majority
The leader proposes the change to its Paxos group and waits for a majority to acknowledge it. Now it is durable.
- Commit wait
The leader waits until its chosen timestamp is definitely in the past, meaning
TT.now().earliesthas passed it. Only then does it release locks and acknowledge the client.
Suppose transaction A commits and the client then starts transaction B anywhere in the world. Because A waited, its timestamp is already past by the time anyone could learn it committed, so B’s timestamp is guaranteed larger. Real-time order becomes timestamp order. The price is roughly the clock uncertainty, a few milliseconds, added to commit latency.
Writes across splits. If a transaction touches rows in several splits, their leaders run two-phase commit: one leader coordinates, all participants lock and vote, and the coordinator commits only if everyone agrees. Each participant is itself a Paxos group, so a participant failing in the middle does not lose the transaction’s state.
Reads without locks
Because every row version has a commit timestamp, a read-only transaction picks a timestamp and reads the latest version at or before it. It needs no locks and cannot block writers.
| Read type | Behaviour | Use when |
|---|---|---|
| Strong read | Returns the latest committed data. The replica confirms it is up to date, possibly asking the leader | Correctness matters most |
| Stale read | Returns data as of a bounded time ago (say 10 seconds) from the nearest replica, no leader round trip | Latency matters and slight staleness is fine |
| Snapshot at timestamp | A consistent view of the database at an exact past time | Audits, debugging, consistent exports |
What TrueTime enables
- External consistency for all transactions.
- Lock-free read-only transactions at a consistent snapshot.
- Atomic schema changes, since a schema update is a transaction with a timestamp that every server honours.
- Historical reads, for free, because versions are kept.
Costs and trade-offs
| Cost | Why |
|---|---|
| Commit latency includes cross-zone Paxos and commit wait | Strong guarantees are paid for in milliseconds |
| Unavailability if a majority cannot be reached | It is a consistency-first system: it stops writes on a split rather than diverge |
| Needs excellent clocks and a private network | Specialised infrastructure, hidden by the managed service |
| Cross-split transactions are slower | Two-phase commit adds round trips, so design data locality |
Do
Choose it when you need relational semantics, global scale and strong consistency together. Design keys so related rows share a split, avoid hotspots from monotonically increasing keys, and use stale reads for latency-sensitive paths that tolerate them.
Don't
Use a globally consistent database for a workload that fits on one Postgres, or for data that can be eventually consistent. You pay in cost and latency for guarantees you do not use.
Takeaway
Spanner’s trick is to stop treating clock error as something to hide and instead measure it, publish it as an interval, and wait it out. Add Paxos replication per key range and multi-version storage, and you get strongly consistent, horizontally scalable SQL. The engineering lesson travels well: when you cannot remove uncertainty, bound it and make correctness depend on the bound.
Frequently asked questions
What is external consistency?
A guarantee stronger than serializability: if transaction A commits before transaction B starts, in real time, every observer sees A's effects before B's, and A's timestamp is lower. The database behaves as if all transactions ran one at a time in real-time order, even across continents.
What is commit wait in Spanner?
After choosing a commit timestamp, the leader waits until that timestamp is definitely in the past according to the clock's uncertainty interval before making the commit visible. It costs a few milliseconds, and it guarantees later transactions anywhere receive larger timestamps.
How does Spanner avoid locking for reads?
Data is multi-versioned with commit timestamps. A read-only transaction picks a timestamp and reads the version of each row at or before it, from any sufficiently up-to-date replica, without taking locks. Strong reads ask the replica to confirm it is current, stale reads allow bounded staleness for lower latency.
Does Spanner break the CAP theorem?
No. It chooses consistency over availability during partitions, and achieves very high availability in practice through engineering: Paxos majorities across zones, private networks and operational discipline. When a majority cannot be reached, the affected split stops accepting writes.

/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.