You declare, you don't command
kubectl writes a record: three replicas of this image. It doesn't start anything. Deleting a pod by hand is just another change the loops will notice and undo.
Who restarts your app when a machine dies at 3 a.m.?
Nobody does, exactly. You write down what should be running, and a handful of small loops keep comparing that with what is running. Each one fixes the one difference it's responsible for, over and over.
One Deployment on three nodes. Each tick, every control loop gets one look at the cluster. Real timeouts are seconds to minutes, not ticks.
Three replicas are running and a pod is deleted. Nobody tells Kubernetes to replace it. The Deployment controller just sees two where the spec says three.
ConvergedDesired and actual state agree: 3 pods, all ready. Every loop is idle.
Converged. Desired and actual state agree: 3 pods, all ready. Every loop is idle.
Replicas · 3
Image
Machines
The loops keep running while this plays. Pause to freeze the cluster.
Pods, as the API server sees them◐ = its node is gone
| pod | image | node | status |
|---|---|---|---|
| web-1 | v1 | node1 | Running · ready |
| web-2 | v1 | node2 | Running · ready |
| web-3 | v1 | node3 | Running · ready |
Counters
3Desired
3Ready
3Serving
0Pending
3Pods created
0Evicted
Sheet 04 · One Deployment, its control loopsThree nodes, three slots eachTicks stand in for timeouts
Three ideas explain most of it
Small loops, one job each
The controller counts pods. The scheduler places them. A kubelet runs what's placed on its node. None of them knows the others exist, and each one only talks to the API server.
Everything is a belief
The control plane only knows what kubelets last reported. A node that dies can't report it, so for a while a pod can be listed as ready on a machine that no longer exists.
Every controller is the same loop
Live: the loop acting now
for (;;) { // every controller is this loop const want = await api.get(deployment); // what you asked for const have = await api.list(pods); // what exists if (have.length < want.replicas) api.create(newPod); if (have.length > want.replicas) api.delete(extraPod);} // scheduler: pod.node == null → api.bind(pod, nodeWithRoom)// kubelet: pod.node == me → start it, report status// node ctrl: no heartbeat → NotReady, then evictLive: this reads the simulation above
$ kubectl get pods -o wide NAME READY STATUS AGE IMAGE NODE web-1 1/1 Running 4t v1 node1 web-2 1/1 Running 4t v1 node2 web-3 1/1 Running 4t v1 node3 $ kubectl get events t1 Normal SuccessfulCreate replicaset/web Created pod web-3 t2 Normal Scheduled web-1 Assigned to node1 t2 Normal Scheduled web-2 Assigned to node2 t2 Normal Scheduled web-3 Assigned to node3 t4 Normal Started web-1 Started container web:v1 t4 Normal Started web-2 Started container web:v1 t4 Normal Started web-3 Started container web:v1
This loop is called reconciliation. It doesn't care how the difference arose. A pod you deleted, a pod lost with its node, or a replica count you raised all look the same to it: fewer pods than asked for.
That's why it's robust. There's no “handle node failure” procedure to get wrong. There's one rule, applied continuously, by a component that can itself crash and restart without losing anything, because all the state lives in the API server.
It's also why it's slow to react. A loop only acts on what it can see, and a missing heartbeat looks the same as a late one until a timeout says otherwise.
It keeps the spec, not your intentions
A dead node takes a while to notice
By default it's marked NotReady after about 40 seconds of silence, and its pods are evicted around five minutes later. Run more than one replica, on more than one node.
It can't make machines
If the pods don't fit, they wait. A cluster autoscaler can add nodes in response to Pending pods, but that's a separate component, and it takes minutes too.
Readiness is what saves you
The bad rollout was harmless only because the new pod never became ready. A pod that passes its readiness check while broken gets traffic, and the rollout carries on happily.
Two predictions
A Deployment runs three replicas. You update it to an image tag that doesn't exist. What do users see?
A node's power is cut. How soon are its pods replaced on other nodes?
What this model leaves out
Watches, not polling
Components don't re-read everything each round. They hold a watch on the API server and get told about changes, then reconcile what changed. The behaviour is the same as the loop here, with far less traffic.
ReplicaSets
A Deployment doesn't own pods directly. It owns one ReplicaSet per pod template, and scales the new one up and the old one down during a rollout. The two are merged into one controller here.
Scheduling and resources
Pods request CPU and memory, and the scheduler filters and scores nodes on those, plus affinity rules, taints and spreading constraints. Here every pod takes one of three slots.
Pods on a lost node
Evicted pods are removed here at once. In a real cluster they can sit in Terminating or Unknown until the node returns or someone force-deletes them. For a StatefulSet, Kubernetes deliberately won't start a replacement until it's sure the old one is gone.
Services and traffic
“Serving” assumes a Service sends traffic to every ready pod. The endpoints controller that does that is one more loop, watching readiness, and isn't drawn.
Connected mechanisms
Containers and InfrastructureDockerWhat actually happens when you run a container?Interactive · 6 min →Distributed SystemsApache KafkaHow does data move between independent systems?Interactive · 8 min →Shipping SoftwareCI/CDHow does code travel from a developer's laptop to production?Interactive · 5 min →