Running Cassandra at Fleet Scale: Hints, Repairs, LWT and Node Replacement
What breaks when you operate thousands of Cassandra nodes: orphan hints, stuck decommissions, flaky lightweight transactions and repairs nobody ran.
On this page
A database that tolerates failures by design still needs a human when the failure handling itself breaks. The interesting story of running Cassandra for years is not data modelling. It is the way small rates of failure multiply across thousands of nodes into a standing on-call burden.
The managed-service shape
At large scale, a dedicated team runs Cassandra as a service for the rest of the company: new features and bug fixes (including upstream contributions), integration with the company’s control plane, configuration, observability and alerting, a four-nines availability target with round-the-clock support, and data-modelling guidance for application teams.
The pieces that make that workable:
| Piece | Role |
|---|---|
| Stateful control plane | Configures and orchestrates thousands of clusters |
| Lifecycle framework | Node selection and replacement, rolling restarts, capacity changes, start and stop |
| Forked clients | Find nodes through service discovery instead of hardcoded addresses, add query fingerprints and observability |
| Service discovery | Notified whenever a node goes up, down or is decommissioned, so clients always see a current list |
Uber has reported tens of millions of queries per second, petabytes of data and tens of thousands of nodes on this model, in clusters of 400 or more nodes across regions.
Failure 1: node replacement that does not finish
Nodes are replaced for hardware failure, fleet changes, topology changes and disaster recovery. Cassandra supports graceful replacement, but at scale three things went wrong: decommissions stuck, new nodes failed to join, and data ended up inconsistent.
A 95 percent success rate sounds fine. At 500 replacements a day it is 25 failures, enough to keep two engineers busy every day. The target for a fleet operation is not “usually works”, it is “so reliable that failures are exceptional”.
The root cause: orphan hints
A hint is a note a node keeps when a peer is unreachable during a write, replayed when the peer returns. The bug was that nodes kept hint files for peers that were no longer in the ring at all. When such a node was decommissioned, it streamed all those orphan hints to its successor, so garbage accumulated across generations to terabytes, and transfers took days.
The fixes were both simple:
- Purge hints for nodes that are not in the ring
Orphans have no one to replay to. Delete them proactively.
- Make the transfer rate adaptive
Adjust the rate limiter dynamically so a hint transfer finishes in hours, not days.
- Expose state, so the control plane can act
A decommission could fail while a rolling restart ran in parallel, and the control plane could not tell. Exposing “decommission failed” and “bootstrap failed” as queryable states let automation retry or clean up correctly.
After the changes Uber reported node replacement reaching 99.99 percent reliability.
Failure 2: lightweight transactions that error every other week
Cassandra’s lightweight transactions (LWT) give a linearizable compare-and-set using a Paxos-based protocol.
INSERT INTO users (id, email) VALUES (42, 'a@example.com') IF NOT EXISTS;
UPDATE accounts SET balance = 90 WHERE id = 7 IF balance = 100;
They cost several round trips, so use them for the few operations that truly need atomic conditional behaviour: unique claims, state transitions, compare-and-set. Everything else should be an ordinary write.
At scale the error rate spiked periodically. Two causes:
- Pending ranges. Several nodes being replaced at once leave token ranges in flux, and LWT needs a clear view of who owns what.
- Stale gossip. When a new node replaced an old one, its gossip path kept resolving the old node’s address and failed with DNS errors, leaving the new node unable to participate until restarted.
Fixing error handling in the gossip code and serialising simultaneous replacements made LWT dependable. The general rule: operations that need coordination expose every sloppy membership change.
Failure 3: replicas that drifted apart
Cassandra replicates each write to several nodes and relies on several mechanisms to converge: read repair, hinted handoff, and anti-entropy repair. Repair compares replicas, using hash trees to find ranges that differ, and streams the missing data.
If repair is slow or skipped, then missed writes stay missed, replicas disagree, and deleted data can resurrect once tombstones are purged on some replicas and not others (the tombstone grace period is the deadline).
Open-source repair tooling usually runs outside the database and triggers repairs through the control plane. Uber chose the opposite: repair as an internal, first-class database process, like compaction.
What to take away, even with a small cluster
| Practice | Why |
|---|---|
| Schedule repair, and alert when it is late | It is the mechanism that keeps replicas equal, and skipping it fails slowly and silently |
| Watch hint volume | Growing hints signal a peer problem, or leftovers from a removed node |
| Replace nodes one at a time | Concurrent membership changes confuse range ownership and gossip |
| Keep LWT use narrow | Several round trips each, and sensitive to cluster instability |
| Expose operation state | Automation can only handle failures it can observe |
| Find clients through service discovery | Replacing a node must not mean redeploying applications |
Do
Automate replacement and repair, measure their success rate and duration, and treat any manual step in a routine operation as a bug to remove.
Don't
Assume “eventually consistent” means “eventually fixed on its own”. Convergence takes working hints, working gossip and completed repairs.
Takeaway
At fleet scale, a database’s hard problems are operational: the leftover hints of departed nodes, the LWT paths that depend on stable membership, and the repairs that only help if they actually run. Make each of them automatic, observable and boring, and the rest of Cassandra’s design, which was built to survive failure, can do its job.
Frequently asked questions
What is hinted handoff in Cassandra?
When a replica is down during a write, another node stores a hint, a note of the missed write, and replays it once the replica returns. It shortens the window of inconsistency after brief outages. Hints are best effort, so they do not replace repair, and unbounded hint buildup is an operational hazard.
Why does Cassandra need repairs?
Replicas can miss writes through dropped messages, expired hints or long outages. Anti-entropy repair compares replicas using Merkle trees and streams missing data so they converge. Without regular repair, deleted data can reappear and replicas drift apart, especially past the tombstone grace period.
What are lightweight transactions?
Conditional writes such as INSERT IF NOT EXISTS or UPDATE ... IF, which use a Paxos-based protocol to give linearizable compare-and-set on a partition. They cost several round trips, so they are far slower than normal writes and should be reserved for the few operations that truly need them.
How do you make node replacement reliable at scale?
Make it a tested, automated workflow with observable states: expose whether a decommission or bootstrap failed, cap and rate-limit streaming of leftover data, and serialise operations that interfere, like rolling restarts. A 95 percent success rate still produces many failures every day on a large fleet.

/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.