---
topic: dev-practices
author: Crashtech Editorial
published: Nov 15, 2026 · read: 4 min
---

Airbnb’s Database-on-Kubernetes Design: Availability Through Safer Replacement

Airbnb’s engineering team explains why it runs one database across three Kubernetes clusters and serializes node replacement to protect quorum.

– –
Three separate server islands remain linked while engineers replace one storage node, illustrating Airbnb’s multi-cluster database design.
Conceptual illustration of serialized node replacement across independent clusters; it does not depict Airbnb’s real infrastructure.

Kubernetes can restart a stateless process without understanding what data it held. A distributed database is different: removing a node changes both the compute topology and the set of replicas carrying durable state. Airbnb’s engineering account makes the operational consequence concrete: replacement has to be serialized, and the incoming storage node needs time to catch up before another replacement begins. [1]

Three clusters, one logical database

Airbnb describes a distributed database cluster spanning three independent Kubernetes clusters, each placed in a different AWS Availability Zone. The database is logically distributed across them; they are not three unrelated database copies. The separation is intended to limit the impact of an infrastructure or configuration failure confined to one zone. Airbnb’s example is a faulty configuration rollout that terminated many VMs in one staging cluster while query-layer nodes in the other two clusters remained available. That is a reported incident from the engineering post, not a guarantee that every two-zone remainder can continue all workloads. [1]

The choice has a cost: teams operate multiple Kubernetes control planes and coordinate changes across them. Airbnb says it uses self-managed Kubernetes clusters in part because that permits rolling back the control plane, whereas a managed control-plane upgrade may not offer the same rollback path. The useful lesson is to map failure domains to the actual dependencies and recovery controls available in your environment before choosing a multi-cluster topology.

Replacement is a state transition, not a pod restart

Airbnb groups replacements into database-initiated changes, proactive infrastructure events such as instance retirement or upgrades, and unplanned failures. For database-initiated changes, its custom Kubernetes operator checks that all nodes are up before deleting a pod. For infrastructure-driven evictions, an admission hook intercepts the eviction, annotates the pod, and lets the operator serialize the operation with database-initiated work. [1]

The safety condition is simple to state: do not start the next planned replacement until the current replacement has rejoined and caught up with the cluster’s current state. Airbnb uses AWS EBS volumes attached to nodes; Kubernetes PersistentVolumeClaims support reattachment to a replacement virtual machine. The article stresses that the storage node still needs catch-up time after reattachment. [1]

planned replacement request
        ↓
operator verifies cluster health
        ↓
replace one node and reattach its EBS volume
        ↓
wait for the replacement to catch up
        ↓
only then begin the next planned replacement

An admission hook does not make simultaneous hardware failure safe. Airbnb explicitly notes that an unplanned infrastructure failure cannot be coordinated in advance; the operator can instead prevent planned maintenance from adding more disruption while the failure is unresolved. That distinction is valuable in runbooks: a controller can serialize actions it sees, but it cannot promise that independent failures will arrive one at a time.

What the design does and does not establish

Airbnb’s post documents a particular database, storage arrangement and operations model. It does not publish a general latency target, a 15-millisecond read policy, follower-read economics, or a universal quorum guarantee. Those details should not be inferred from the architecture diagram. Kubernetes disruption controls can help limit voluntary interruptions, but node failures, application-level replication, storage reattachment and control-plane recovery remain separate concerns. [2]

For a team evaluating a similar design, the practical questions are:

  1. Which maintenance events can your operator observe and serialize?
  2. What health signal proves the replacement has rejoined and caught up?
  3. Which storage resources are zone-bound, and how long does reattachment take in your environment?
  4. What happens when hardware fails while planned maintenance is waiting?
  5. Can the control plane itself be recovered or rolled back within the database’s availability objective?

Answer those with failure drills and measured behavior from your own system. Airbnb’s architecture is a useful case study, not a turnkey recipe.

Production Best Practices for Stateful Workloads

Do

  • Define an explicit health and catch-up gate before starting the next planned stateful replacement.
  • Coordinate database initiated replacement with infrastructure initiated evictions so two controllers do not act independently.
  • Test storage detach and reattachment behavior in the same zones and Kubernetes version you operate.

Don't

  • Do not assume a replacement pod is ready just because Kubernetes reports it scheduled.
  • Do not treat a multi-cluster layout as protection against every database, network or control-plane failure.
  • Do not run planned maintenance through a second controller that bypasses the database-aware replacement gate.

Airbnb’s account shows how database-aware orchestration can make planned replacement safer on Kubernetes. Its central operational rule is to let state recovery finish before beginning the next planned change, while treating unplanned failures and control-plane recovery as distinct risks. For related mechanics, see our explainers on the Kubernetes pod lifecycle and availability and failure domains.

Advertisement

Frequently asked questions

Why does Airbnb place its distributed database across multiple Kubernetes clusters?

Airbnb describes three independent clusters, each in a different AWS Availability Zone, to limit the blast radius of a faulty deployment or infrastructure problem. This adds operational complexity and is not a universal recommendation.

How does Airbnb coordinate planned database node replacement?

Its custom Kubernetes operator checks that nodes are healthy before database initiated replacement. An admission hook intercepts infrastructure initiated evictions, records them for the operator, and lets the operator serialize the work.

What happens after an Airbnb database node is replaced?

Airbnb says its EBS volume can be reattached through Kubernetes PersistentVolumeClaims, then the replacement storage node is allowed to catch up with cluster state before another node is replaced.

Does this design prevent every outage?

No. Airbnb distinguishes coordinated maintenance from unplanned hardware failure, which cannot be serialized in advance. Separate clusters reduce some shared failure risks but do not guarantee availability or eliminate database-level failure modes.

Sources & further reading

/* Comments */