---
topic: system-design
author: Crashtech Editorial
published: Nov 14, 2026 · read: 5 min
---

Running Cassandra at Fleet Scale: Hints, Repairs, LWT and Node Replacement

What breaks when you operate thousands of Cassandra nodes: orphan hints, stuck decommissions, flaky lightweight transactions and repairs nobody ran.

– –

A database that tolerates failures by design still needs a human when the failure handling itself breaks. The interesting story of running Cassandra for years is not data modelling. It is the way small rates of failure multiply across thousands of nodes into a standing on-call burden.

The managed-service shape

At large scale, a dedicated team runs Cassandra as a service for the rest of the company: new features and bug fixes (including upstream contributions), integration with the company’s control plane, configuration, observability and alerting, a four-nines availability target with round-the-clock support, and data-modelling guidance for application teams.

The pieces that make that workable:

PieceRole
Stateful control planeConfigures and orchestrates thousands of clusters
Lifecycle frameworkNode selection and replacement, rolling restarts, capacity changes, start and stop
Forked clientsFind nodes through service discovery instead of hardcoded addresses, add query fingerprints and observability
Service discoveryNotified whenever a node goes up, down or is decommissioned, so clients always see a current list

Uber has reported tens of millions of queries per second, petabytes of data and tens of thousands of nodes on this model, in clusters of 400 or more nodes across regions.

Failure 1: node replacement that does not finish

Nodes are replaced for hardware failure, fleet changes, topology changes and disaster recovery. Cassandra supports graceful replacement, but at scale three things went wrong: decommissions stuck, new nodes failed to join, and data ended up inconsistent.

Do the arithmetic on rare failures

A 95 percent success rate sounds fine. At 500 replacements a day it is 25 failures, enough to keep two engineers busy every day. The target for a fleet operation is not “usually works”, it is “so reliable that failures are exceptional”.

The root cause: orphan hints

A hint is a note a node keeps when a peer is unreachable during a write, replayed when the peer returns. The bug was that nodes kept hint files for peers that were no longer in the ring at all. When such a node was decommissioned, it streamed all those orphan hints to its successor, so garbage accumulated across generations to terabytes, and transfers took days.

The fixes were both simple:

  1. Purge hints for nodes that are not in the ring

    Orphans have no one to replay to. Delete them proactively.

  2. Make the transfer rate adaptive

    Adjust the rate limiter dynamically so a hint transfer finishes in hours, not days.

  3. Expose state, so the control plane can act

    A decommission could fail while a rolling restart ran in parallel, and the control plane could not tell. Exposing “decommission failed” and “bootstrap failed” as queryable states let automation retry or clean up correctly.

After the changes Uber reported node replacement reaching 99.99 percent reliability.

Failure 2: lightweight transactions that error every other week

Cassandra’s lightweight transactions (LWT) give a linearizable compare-and-set using a Paxos-based protocol.

INSERT INTO users (id, email) VALUES (42, 'a@example.com') IF NOT EXISTS;
UPDATE accounts SET balance = 90 WHERE id = 7 IF balance = 100;

They cost several round trips, so use them for the few operations that truly need atomic conditional behaviour: unique claims, state transitions, compare-and-set. Everything else should be an ordinary write.

At scale the error rate spiked periodically. Two causes:

  • Pending ranges. Several nodes being replaced at once leave token ranges in flux, and LWT needs a clear view of who owns what.
  • Stale gossip. When a new node replaced an old one, its gossip path kept resolving the old node’s address and failed with DNS errors, leaving the new node unable to participate until restarted.

Fixing error handling in the gossip code and serialising simultaneous replacements made LWT dependable. The general rule: operations that need coordination expose every sloppy membership change.

Advertisement

Failure 3: replicas that drifted apart

Cassandra replicates each write to several nodes and relies on several mechanisms to converge: read repair, hinted handoff, and anti-entropy repair. Repair compares replicas, using hash trees to find ranges that differ, and streams the missing data.

If repair is slow or skipped, then missed writes stay missed, replicas disagree, and deleted data can resurrect once tombstones are purged on some replicas and not others (the tombstone grace period is the deadline).

Open-source repair tooling usually runs outside the database and triggers repairs through the control plane. Uber chose the opposite: repair as an internal, first-class database process, like compaction.

A dedicated thread pool resource isolation
Repair work has its own scheduler threads so it does not starve normal traffic, nor get starved by it.
Replicated repair state no external brain
A replicated table records each node’s repair status and history, so any node can see what ran and what is due.
Ordered orchestration coverage guarantee
The scheduler picks the node that repaired longest ago and walks every table and token range until all are covered.
No control-plane dependency less toil
Repair continues when the orchestrator is down, and operators stop hand-scheduling jobs.

What to take away, even with a small cluster

PracticeWhy
Schedule repair, and alert when it is lateIt is the mechanism that keeps replicas equal, and skipping it fails slowly and silently
Watch hint volumeGrowing hints signal a peer problem, or leftovers from a removed node
Replace nodes one at a timeConcurrent membership changes confuse range ownership and gossip
Keep LWT use narrowSeveral round trips each, and sensitive to cluster instability
Expose operation stateAutomation can only handle failures it can observe
Find clients through service discoveryReplacing a node must not mean redeploying applications

Do

Automate replacement and repair, measure their success rate and duration, and treat any manual step in a routine operation as a bug to remove.

Don't

Assume “eventually consistent” means “eventually fixed on its own”. Convergence takes working hints, working gossip and completed repairs.

Takeaway

At fleet scale, a database’s hard problems are operational: the leftover hints of departed nodes, the LWT paths that depend on stable membership, and the repairs that only help if they actually run. Make each of them automatic, observable and boring, and the rest of Cassandra’s design, which was built to survive failure, can do its job.

Advertisement

Frequently asked questions

What is hinted handoff in Cassandra?

When a replica is down during a write, another node stores a hint, a note of the missed write, and replays it once the replica returns. It shortens the window of inconsistency after brief outages. Hints are best effort, so they do not replace repair, and unbounded hint buildup is an operational hazard.

Why does Cassandra need repairs?

Replicas can miss writes through dropped messages, expired hints or long outages. Anti-entropy repair compares replicas using Merkle trees and streams missing data so they converge. Without regular repair, deleted data can reappear and replicas drift apart, especially past the tombstone grace period.

What are lightweight transactions?

Conditional writes such as INSERT IF NOT EXISTS or UPDATE ... IF, which use a Paxos-based protocol to give linearizable compare-and-set on a partition. They cost several round trips, so they are far slower than normal writes and should be reserved for the few operations that truly need them.

How do you make node replacement reliable at scale?

Make it a tested, automated workflow with observable states: expose whether a decommission or bootstrap failed, cap and rate-limit streaming of leftover data, and serialise operations that interfere, like rolling restarts. A 95 percent success rate still produces many failures every day on a large fleet.

Sources & further reading

/* Comments */