Distributed Locks Done Safely: Leases, Fencing Tokens and When to Avoid Them
A lock that expires while its holder is paused is not a lock. Why timeout locks need fencing tokens, and the alternatives that avoid locking.
On this page
“Just grab a lock in Redis” is the most casually dangerous sentence in backend engineering. On one machine a lock is simple. Between machines, there is a gap between “I hold the lock” and “my write arrives”, and a lot can happen in it.
What people use them for
| Use | Lock means |
|---|---|
| Leader election | Exactly one node acts as leader at a time |
| Scheduled job runs once | Only one worker executes the task |
| Resource allocation | One process uses a device, file or socket at a time |
| Inventory or balance updates | One writer changes the record at once |
| Coordinating services | Related changes happen in a controlled order |
| Session updates | A user session is modified by one server at a time |
All six sound the same and are not the same. Ask first: what happens if two nodes both believe they hold it?
Two reasons to lock
How a timeout lock breaks
To avoid deadlock when a holder crashes, locks carry a lease: they expire after a set time. That is necessary, and it opens the hole.
Client A acquires lock (lease 10s)
Client A pauses 15s (GC pause, stalled VM, network stall)
Lease expires
Client B acquires the lock, writes to storage
Client A wakes up, still believes it holds the lock, writes to storage
-> both wrote; the later arrival wins and may overwrite BClient A did nothing wrong by its own reasoning: it checked the lock, then wrote. The pause happened between the check and the write, and nothing in the lock can prevent that. Making the lease longer reduces the odds but never to zero, and clocks drifting or jumping can shorten a lease unexpectedly.
A lock whose safety depends on “the lease is longer than any pause” is a bet about the environment. Garbage collection, virtual machine migration and network delay all break that bet occasionally, and rare events happen constantly at scale.
The fix: fencing tokens
Make the resource reject stale holders, since the lock cannot.
- The lock service issues a number
Every successful acquisition returns a token that strictly increases: 33, then 34, then 35.
- The client sends the token with every write
The write to storage includes the token it was granted.
- The storage rejects older tokens
It remembers the highest token it has accepted and refuses anything lower.
A gets token 33, pauses
B gets token 34, writes (storage records 34)
A wakes, writes with 33 -> rejected: 33 < 34This requires a lock service that can generate monotonic tokens, such as a consensus-backed store like ZooKeeper or etcd (revision numbers serve well), and a resource that checks them. A plain single-node Redis SET NX PX does not provide tokens, which is why it is appropriate for efficiency locks and not for correctness.
Often, skip the lock
The strongest designs move the guarantee into one atomic operation the data store already provides.
| Instead of locking… | Use |
|---|---|
| Read, modify, write a row | Optimistic concurrency: UPDATE ... WHERE id = ? AND version = ?, retry on zero rows |
| ”Only one job runs” | A unique constraint or insert-if-absent on a job key |
| ”Process each event once” | Idempotency keys and a processed-events table |
| ”Serialise changes per account” | Route all writes for an account to one queue partition or one worker |
| Inventory decrement | UPDATE stock SET qty = qty - 1 WHERE sku = ? AND qty > 0 |
| Leader election | A consensus system’s built-in primitive, not a hand-built lock |
-- no distributed lock needed: the database makes this atomic
UPDATE inventory SET qty = qty - 1 WHERE sku = 'A1' AND qty > 0;
-- rows affected = 0 means sold outIf you do need a lock
Do
Decide whether it is for efficiency or correctness. For correctness, use a consensus-backed service, pass fencing tokens to the resource, keep the critical section short, and make the protected operation idempotent.
Don't
Take a Redis lock, do a slow operation, then write to a database that cannot check the lock. You have built something that works in testing and fails under a garbage-collection pause.
Takeaway
A distributed lock cannot make a paused process stop. So either accept occasional duplicates and use a simple lease, or make the resource enforce order with fencing tokens. Most of the time the better answer is to drop the lock and lean on an atomic operation the database already guarantees: a version check, a unique constraint, a conditional update, or a single writer per key.
Frequently asked questions
What is a distributed lock used for?
Ensuring only one node at a time performs an action across a cluster: electing a leader, running a scheduled job once, updating a shared resource, or serialising changes to one entity. Locks used for efficiency tolerate rare double execution. Locks used for correctness must prevent it absolutely.
Why can a lock with a timeout still allow two holders?
A client can acquire a lock, pause for longer than the timeout (garbage collection, a stalled network, a busy machine), then resume believing it still holds the lock while another client has since acquired it. Both now act. The lock's expiry cannot stop the paused one.
What is a fencing token?
A number that increases with every lock grant. The client sends it with every write, and the storage system rejects any write carrying a token older than one it has already seen. A stale holder that wakes up late is then refused by the resource itself.
Can I avoid distributed locks entirely?
Often. Use optimistic concurrency with version columns, idempotent operations, a single-writer queue per key, or a database unique constraint. These make correctness depend on the data store's atomic operations instead of on timing assumptions between machines.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.