Why Raft Consensus Prevents Split-Brain

When a primary database node fails, a secondary must take over. In active-passive failover, humans detect the failure and promote the standby—hours of unavailability and a race condition if the “failed” primary network-partitions and resurrects. Raft automates this: a new leader is elected in milliseconds by a quorum, and the cluster stays coherent because only one partition can ever reach majority. A write is durably committed to the cluster only after replication to a majority, making divergence impossible.

Left: split-brain risks dual primaries with diverged data. Right: Raft quorum prevents this automatically.

The Split-Brain Trap

Active-passive failover seems simple: one primary, one (or more) standby replicas. Writes go to the primary; reads can use standbys. When the primary crashes, promote a standby.

But networks don’t just fail cleanly. They partition. Imagine a five-node cluster: one primary and four standbys. A network cable loosens. The primary is now unreachable from the standbys, but the primary still thinks it is primary—its own disk and memory are fine. The standbys don’t hear heartbeats and assume the primary is dead. After a configurable timeout, they elect one of themselves as the new primary.

Now there are two nodes both accepting writes: the old primary in one partition, the new primary in another. Both write data at the same time. When the network heals, neither node can know whose writes are “correct”—consistency is shattered.

This is split-brain: two nodes both believe they are primary, diverging data, with no automatic way to recover. A human must choose which node’s data to keep and which to discard. Worse, the detection and manual failover take minutes or hours—during which the cluster is either unavailable or serving stale data.

Active-passive systems prevent split-brain only through external consensus: a human operator reads monitoring data and makes a decision. That decision is a bottleneck and a source of human error.

How Raft Prevents It Automatically

Raft’s insight is deceptively simple: a leader must have buy-in from the majority before it can commit any write.

When the primary fails, no node in the minority partition can convince a majority to vote for it as the new leader. A candidate needs votes from a majority of the cluster to become leader. In a five-node cluster, a majority is 3. If the partition is 2-3, the smaller partition’s 2 nodes cannot reach 3 votes no matter what they do—they cannot elect a leader, and they automatically block writes. The 3-node partition elects a leader and continues serving.

The mathematics are airtight: if you partition N nodes into two groups, at most one group can be > N/2. You cannot have two groups of 3 in a 5-node cluster. The moment the network heals, the follower nodes in the minority partition will see the term and log of the majority’s leader, adopt that as truth, and resync.

This is why Raft is at the heart of etcd (Kubernetes’s configuration store) and CockroachDB (a distributed SQL database). Both need automatic, trustworthy failover with zero manual intervention.

The Mechanism: Terms, Timeouts, Votes, and Logs

Raft coordinates leaders through three mechanisms: terms, randomized election timeouts, and log-based quorum commits.

Terms: Logical Clocks

Every node tracks a current term, a monotonically increasing integer. Each term is a potential election round. A leader broadcasts its term to all followers via heartbeats (empty append-log messages) at regular intervals. If a heartbeat is late or missing, followers assume the leader is dead.

When a follower detects a leader failure (heartbeat timeout), it increments its term and becomes a candidate. It broadcasts a RequestVote RPC asking peers to vote for it as the new leader for this term. Any node that receives a RequestVote with a higher term immediately adopts that term, stepping down from leadership if necessary.

Randomized Election Timeouts

If every follower became a candidate immediately upon heartbeat timeout, they would all request votes simultaneously, splitting the vote and deadlocking. Raft uses randomized election timeouts: each node waits a random duration (e.g., 150–300 ms) before timing out. The first candidate to wake up sends RequestVote messages before competitors do. If it wins a majority of votes, it becomes leader for that term. The new leader broadcasts heartbeats, resetting everyone’s election timeout, and the cycle repeats.

This randomization is critical: with 5 nodes, one will almost always wake up slightly ahead of the others, win votes, and prevent dueling candidates.

Quorum Commits

A write does not become durable in Raft just because the leader logged it. The leader replicates the log entry to all followers via AppendEntries RPCs. A follower acknowledges when it has durably written the entry. The leader counts acknowledges. Only when a majority (quorum) has acknowledged the entry does the leader mark it as committed.

Once an entry is committed, the leader can apply it to the state machine and return success to the client. It also tells followers which entries are committed in the next heartbeat. Followers apply committed entries in order, guaranteeing that every node’s state machine converges on the same deterministic history.

Why This Prevents Split-Brain

When a partition occurs:

1. The minority partition’s candidates request votes, but there are not enough nodes to reach a quorum. If the partition has 2 nodes, it needs 3 votes out of 5 total—impossible. The candidates give up and remain followers, blocking all writes (returning “leader not available”).

2. The majority partition’s candidate reaches quorum and becomes leader. It can immediately start replicating writes, and new writes commit as soon as a quorum acknowledges them.

3. Nodes in the minority partition have no leader, so client writes fail with a clear error, not a silent divergence. Operators know to route traffic to the majority partition.

4. When the network heals, minority followers see the majority leader’s heartbeat with a higher term, adopt its log, and resync.

Election: candidate requests votes from all. Quorum (3 of 5) is needed to become leader. Log replication: entries commit only when acknowledged by a majority. Any network partition can only have one majority.

What Raft Guarantees (and What It Doesn’t)

Raft guarantees safety: as long as a quorum of nodes is reachable, the cluster will elect exactly one leader per term, and all committed entries will be replicated to a majority and survive failures.

Raft does not guarantee availability: if a partition is smaller than N/2, it cannot elect a leader or commit writes. This is the right trade-off for a database or configuration store—losing some availability is far preferable to risking data corruption or divergence.

Raft also assumes that nodes can communicate reliably enough to detect failure (i.e., the heartbeat timeout must be much longer than the longest network delay). If heartbeat timeouts are set too aggressively, network jitter can cause frivolous elections.

When NOT to Use Raft

Raft is powerful but not free. Each commit requires RPCs to a majority, and each election can cause a brief unavailability window (typically < 1 second). For systems that:

• Do not need strong consistency (e.g., a read-heavy cache with eventual consistency), Raft’s cost outweighs its benefit. Gossip protocols or simple replication lag can be enough.
• Have extremely tight latency budgets and can tolerate stale reads, a simpler primary-backup system with manual failover might be preferable.
• Operate in wide-area networks with high latency (e.g., multi-region), Raft’s heartbeat and election timeouts can thrash. A leasing or epoch-based scheme may fit better.

But for any distributed system where consistency and availability must coexist—Kubernetes state, a distributed database, a distributed lock service—Raft is the industrial standard. It is used by etcd, CockroachDB, Consul, TiDB, and dozens of other production systems.

The Takeaway

Split-brain is not a hypothetical: it happens every time a network partitions. Active-passive failover leaves humans in the loop. Raft automates consensus: a leader is elected by quorum within milliseconds, a write is committed only after replication to a majority, and the mathematics guarantee that two partitions cannot both claim primacy. This is why Raft has become the foundation of modern distributed systems, replacing manual failover and external consensus mechanisms.

Production Examples

Kubernetes (etcd): Every Kubernetes cluster stores its state (pod definitions, config, secrets) in etcd, a Raft-based key-value store. When the etcd leader fails, a new leader is elected within seconds, and Kubernetes continues scheduling and managing pods without manual intervention.

CockroachDB: A distributed SQL database that replicates every range (shard) of data across multiple nodes using Raft. A write in CockroachDB is committed to disk on a quorum of replicas before being acknowledged to the client, guaranteeing that durability survives any single-node failure.

Consul: HashiCorp’s distributed service mesh uses Raft for leader election and configuration consensus across data centers, enabling automatic service discovery and configuration without manual coordination.