Keeping a Million Clocks Honest: From NTP to Precision Time Protocol
Why drifting clocks break distributed systems, how NTP and leap-second smearing work, and why hardware timestamps take sync to nanoseconds.
On this page
A clock that is visibly broken gets fixed. A clock that is confidently wrong by 40 milliseconds just reorders your transactions. At scale the second kind is the dangerous one, because every machine believes it knows what time it is.
Why clocks drift
Every machine counts time with a small crystal oscillator: the clock’s heartbeat. Its frequency shifts with temperature, age and manufacturing tolerance. A tiny error rate compounds to seconds per day, so every clock needs periodic correction against a better reference.
The cost of being wrong scales with what you do with timestamps: log correlation, cache expiry, certificate validity, scheduled jobs, and above all ordering writes across machines. If server A stamps a write at 10:00:00.003 and server B stamps a later write at 10:00:00.001 because its clock runs behind, “last write wins” now keeps the wrong one.
NTP: ask, measure, adjust
Network Time Protocol is the workhorse. Clients periodically ask servers for the time, estimate network delay from the exchange, and nudge their clocks. Servers are arranged in strata, with the most accurate at the top.
| Stratum | What it is |
|---|---|
| 0 | Reference clocks: GNSS satellites (GPS, Galileo, GLONASS) and atomic clocks |
| 1 | Servers directly attached to a stratum 0 source |
| 2 | Servers synchronising to stratum 1 |
| 3 and below | Everyone else, as many layers as needed to scale |
The assumption inside NTP’s delay estimate is that the path to the server and the path back take the same time. When that holds, error is small. When it does not, the clock is wrong by half the difference, with no way to notice.
Large operators run their own fleets of time servers and favour modern daemons. Meta has described choosing chrony over ntpd after testing showed better accuracy and scalability and lower memory use, and reported improving accuracy from around 10 milliseconds to around 100 microseconds with that service.
Leap seconds and the smear
Earth’s rotation is uneven, so civil time occasionally adds or removes a second. To a human it is nothing. To a server that assumes time only moves forward, a repeated second can cause missed timers, duplicate timestamps and crashes.
Smearing spreads the second across hours in tiny increments, so the clock never jumps. Google documents a smear on its public NTP service, and Meta has described applying the smear at stratum 2 so downstream servers never see a leap second at all.
If some of your servers use a smeared time source and others step the leap second, their clocks disagree by up to half a second for hours. Pick one policy for the whole fleet and enforce it.
Why PTP goes further
NTP’s remaining error comes from things in the path: software scheduling, network drivers, and queues in switches. Precision Time Protocol attacks each of them.
- Hardware timestamping
The network card stamps the packet at the moment it leaves or arrives, instead of the operating system stamping it some unpredictable microseconds later. Software and driver delay disappear from the measurement.
- Transparent clocks
Each switch measures how long a PTP packet sits inside it and writes that into the packet. The client subtracts it, so queuing delay stops contaminating the result.
- Better symmetry
Delay estimation assumes the forward and return paths are equal. Hardware stamps and transparent switches make that assumption far closer to true.
The result is synchronisation in nanoseconds on a well-built network, roughly a hundred times better than NTP, according to Meta’s account.
What precise time buys
Meta’s own example is the classic read-your-writes problem. You post an update, then your next request lands on a different server that has not seen it yet. The old fix was to ask several servers and wait for a majority, which costs extra round trips and compute. With tightly synchronised clocks, the read can simply wait until the server’s time passes the timestamp of your write. One request, no polling.
Other uses it cites include event tracing, cache invalidation, privacy-violation detection and latency compensation in immersive applications.
Time as a range, not a number
Even excellent synchronisation leaves some error. The honest API admits it.
now() -> [earliest, latest] # true time lies inside this windowIf two events’ windows do not overlap, their order is certain. If they overlap, you wait out the uncertainty or treat them as concurrent. Google’s TrueTime for Spanner and Meta’s fbclock both work this way. It is the same idea as an error bar: report the measurement and how much to trust it.
What to do with this
Do
Use a good time daemon (chrony), point it at multiple trusted sources, monitor offset and alert on drift, choose one leap-second policy fleet-wide, and prefer logical clocks or versions for ordering writes unless you truly have bounded-error time.
Don't
Resolve conflicts with “latest timestamp wins” on ordinary cloud VMs and treat the result as authoritative. A few milliseconds of skew is enough to discard the newer write.
Takeaway
Time in a distributed system is an estimate with an error bar. NTP gets you milliseconds and hundreds of microseconds, PTP with hardware timestamps and transparent switches gets you nanoseconds, and a window-of-uncertainty API lets code stay correct anyway. Know your error bar before you build anything that depends on “which came first”.
Frequently asked questions
Why do computer clocks drift?
Each machine keeps time with a small crystal oscillator whose frequency varies slightly with temperature, age and manufacturing differences. Even tiny errors add up to seconds per day, so clocks must be corrected regularly against a reference. Without it, two servers disagree about the current time.
What is a leap second and why is smearing used?
A leap second is an extra second inserted occasionally to keep civil time aligned with Earth's rotation. A sudden jump can break software that assumes time never repeats. Smearing spreads the adjustment across several hours in tiny increments so clocks never see a discontinuity.
How is PTP more accurate than NTP?
Precision Time Protocol timestamps packets in network hardware, so software and driver delays drop out of the measurement. Network switches acting as transparent clocks record how long each packet waited inside them. Together this reaches sub-microsecond accuracy where NTP over a typical network manages milliseconds to microseconds.
What does a window of uncertainty API return?
Instead of a single timestamp it returns a time range guaranteed, within a stated confidence, to contain the true time. Callers can then reason safely: if two events' windows do not overlap, their order is certain. Google's TrueTime and Meta's fbclock are examples.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.