What Really Happens After kubectl apply
Follow one Deployment from kubectl apply to a running pod: API server, etcd, controllers, scheduler and kubelet each own one step of the chain.
On this page
People describe Kubernetes as “it runs my containers”. It does not run anything on command. It records what you want and then a chain of small programs keeps making reality match the record. Once you see the chain, debugging a stuck deployment stops being guesswork.
The two halves of the cluster
The control plane makes decisions: the API server, etcd, the scheduler and the controller manager. Nodes do the work: each runs a kubelet (talks to the container runtime), a container runtime, and kube-proxy (programs the networking rules for Services).
Follow one Deployment
- kubectl sends your YAML to the API server
The API server is the front door and the only component that talks to etcd. It authenticates you, authorizes the request, runs admission checks, validates the object, and persists it. At this point nothing is running. A Deployment object simply exists in the store.
- The Deployment controller notices
Controllers watch the API server for changes. The Deployment controller sees a Deployment with no ReplicaSet and creates one, carrying the pod template and the replica count.
- The ReplicaSet controller creates Pods
It sees a ReplicaSet wanting three pods and finding none, so it creates three Pod objects. They sit in state Pending with no node assigned.
- The scheduler assigns nodes
The scheduler watches for pods without a node. For each one it filters nodes that cannot fit (resource requests, taints, affinity) and scores the rest, then writes the chosen node name back to the API server. It never touches a container.
- The kubelet starts the containers
The kubelet on the chosen node sees a pod assigned to it, pulls the image, asks the container runtime to start the containers, and reports status back. Only now does anything actually run.
Notice what is missing: no component calls another directly. Everything goes through the API server and the stored state. That indirection is the architecture.
A Pending pod with no node means the scheduler could not place it: not enough CPU or memory, an unsatisfiable node selector, or a taint with no toleration. kubectl describe pod shows the scheduler’s reason in the events. A pod with a node but stuck in ContainerCreating points to the kubelet side: image pull failures, volume mounts, or network setup.
The objects you touch most
| Object | What it is for |
|---|---|
| Pod | One or more containers sharing a network namespace and volumes. The smallest deployable unit. |
| Deployment | Declarative rollout and rollback for stateless pods, via ReplicaSets. |
| Service | A stable virtual IP and DNS name that selects pods by label, so callers survive pod churn. |
| Volume / PersistentVolume | Storage that outlives a container restart, and with a PersistentVolume, outlives the pod. |
| Horizontal Pod Autoscaler | Adjusts replica count from CPU, memory or custom metrics. |
The HPA is the same pattern again: a controller reads a metric, compares it to a target, and edits the replica count. Your Deployment controller then reacts to the new number.
Reconciliation is the whole trick
Every controller runs the same loop: read desired state, read actual state, act on the difference, repeat. Kill a pod and the ReplicaSet controller sees two of three and creates one more. Drain a node and the pods are recreated and rescheduled elsewhere. Nobody wrote recovery code for your app. Recovery is the default behaviour of the loop.
Do
Debug by following the chain: object exists? controller created the child? scheduler assigned a node? kubelet started it? The first broken link is your bug.
Don't
Delete and recreate things at random. Controllers will just recreate them from the same broken spec, and you lose the events that explain what failed.
Takeaway
Kubernetes is not a launcher, it is a set of independent control loops converging on a declared state. Learn who owns each step (API server stores, controllers create objects, scheduler picks nodes, kubelet runs containers) and every symptom maps to one place to look.
Frequently asked questions
What is the role of etcd in Kubernetes?
etcd is the consistent key-value store that holds the entire desired and observed state of the cluster. Only the API server talks to it. Every other component reads and writes state through the API server, which is why losing etcd without a backup means losing the cluster's brain.
Who actually creates the pods when I apply a Deployment?
Not the API server. The Deployment controller creates a ReplicaSet, and the ReplicaSet controller creates the Pod objects. The scheduler then assigns each pod to a node, and the kubelet on that node starts the containers. Each component does one narrow job.
What does the scheduler do and not do?
The scheduler only decides which node a pending pod should run on, based on resource requests, constraints and affinity rules, then records that choice. It never starts containers. The kubelet on the chosen node notices the assignment and does the actual work.
What is a reconciliation loop?
A controller repeatedly compares desired state (what you declared) with observed state (what exists) and acts to close the gap. If a pod dies, the loop notices the count is below the desired replicas and creates another. This is why Kubernetes self-heals without anyone running a command.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.