---
topic: devops
author: Crashtech Editorial
published: Nov 17, 2026 · read: 3 min
---

Kubernetes as a System of Promises: Reconciliation Loops and Desired State

Understand Kubernetes control plane architecture: declarative desired state, etcd, kube-apiserver, and controller reconciliation loops.

– –

Most engineers learn infrastructure through imperative scripts: “Spin up three virtual machines, install Docker, copy the binary, run systemctl start”.

The fatal flaw of imperative scripts is that they describe a procedure, not an outcome. If step three crashes halfway through, the script cannot recover. Worse, if a virtual machine terminates at 3:00 AM, the script is already dead and gone; it cannot detect that the system has drifted away from the intended operational state.

Kubernetes abandoned the procedural model entirely. It is designed around a single architectural metaphor: Kubernetes is a system of promises.

You do not tell Kubernetes how to deploy software. You give Kubernetes a declarative promise representing your Desired State (spec: replicas: 3). The internal machinery of Kubernetes then runs an infinite Reconciliation Loop, constantly comparing the physical reality of the cluster (Current State) against your declared promise, executing corrective mutations until the difference is zero.

This guide explores the foundational architecture of the Kubernetes control plane: kube-apiserver, etcd, informers, and the level-triggered reconciliation loops that power modern cloud orchestration.


Imperative vs Declarative Orchestration

Imperative (Procedural):
[ Human / Script ] ──► "Create VM 1" ──► "Create VM 2" ──► "Create VM 3"
(If VM 2 crashes hours later, nobody knows)

Declarative (Kubernetes):
[ Human ] ──► "Ensure exactly 3 healthy replicas exist at all times"
                    │
                    ▼
[ Reconciliation Loop ] ◄── (Watches cluster state 24/7)
  - Sees 2 running pods? Spins up 1.
  - Sees 4 running pods? Terminates 1.
  - Sees healthy 3 pods? Does nothing.

Kubernetes as a System of Promises: Reconciliation Loops


The Control Plane Architecture

The Kubernetes control plane coordinates this self-healing loop through four core decoupled binaries:

1. kube-apiserver (The Gateway)

The API server is the only component in the entire cluster that communicates directly with the persistent database. Every other component (the scheduler, controllers, node kubelets) communicates strictly via REST calls to kube-apiserver. It validates authentication, applies authorization policies (RBAC), and runs admission webhooks.

2. etcd (The Source of Truth)

etcd is a strongly consistent, distributed key-value store implementing the Raft consensus algorithm. It holds the complete declared desired state and observed status of every resource in the cluster. It provides a multi-version concurrency control (MVCC) watch API that streams changes to listeners.

3. kube-scheduler (The Placement Engine)

The scheduler watches for newly created pods that lack an assigned node (spec.nodeName == ""). It filters available nodes based on resource requests (CPU, memory), taints, tolerations, and affinity rules, assigning the pod to the optimal physical worker.

4. kube-controller-manager (The Brain)

The controller manager bundles dozens of independent reconciliation loops into a single process: the DeploymentController, ReplicaSetController, NodeController, and EndpointSliceController.


Anatomy of a Reconciliation Loop

How does a controller detect that a container died without drowning kube-apiserver in polling requests?

Modern Kubernetes controllers use Reflectors, Informers, and WorkQueues:

[ etcd ] ──(Watch Stream)──► [ Informer (Local Cache) ]
                                    │
                                    │ Event: OnAdd / OnUpdate / OnDelete
                                    ▼
                             [ WorkQueue (Deduplicated Keys) ]
                                    │
                                    ▼
                         [ Reconcile(Request) ]
                           1. Fetch Desired State (spec)
                           2. Fetch Current State (status)
                           3. Compute Delta
                           4. Actuate Fix (API Mutation)

Level-Triggered vs Edge-Triggered

Traditional event-driven systems are edge-triggered: they react to the transition itself (“Network packet arrived”). If an edge event is dropped due to network jitter, the system remains broken forever.

Kubernetes reconciliation is level-triggered: the controller does not care how many events occurred. It examines the current level of the system right now. Even if five network messages were lost, the next reconciliation cycle inspects the actual world and restores balance.


The Node Agent: kubelet

Worker nodes run a dedicated daemon called kubelet. The kubelet is a local reconciliation loop running on physical hardware:

  1. It registers the node with kube-apiserver.
  2. It watches for PodSpecs assigned to its node name.
  3. It calls the Container Runtime Interface (CRI) (e.g., containerd) to pull images and start Linux cgroups and namespaces.
  4. It calls the Container Network Interface (CNI) (e.g., Cilium, Calico) to assign pod IP addresses.
  5. It monitors container health checks (liveness and readiness probes), reporting status back to the API server.

Architectural Lessons for Distributed Systems

  1. Declarative APIs Scale Better: When systems accept desired outcomes instead of execution instructions, clients are insulated from failure recovery mechanics.
  2. Decouple the Brain from State: Separating the stateless controller loop (kube-controller-manager) from the consensus database (etcd) allows controllers to crash and reboot without losing cluster state.
  3. Design for Continuous Self-Healing: Assume that servers will crash, disks will fill, and networks will partition. An infinite reconciliation loop is the most resilient antidote to operational entropy.
Advertisement

/* Comments */