Node Drain
How do you take a Node out of service without dropping your apps? The drain cordons it first, so the Scheduler skips it unless a Pod tolerates the unschedulable taint, then evicts the Pods through the Eviction API rather than deleting them. A PodDisruptionBudget holds an eviction back with a 429 until a replacement is Ready elsewhere. A drain leaves running DaemonSet Pods in place, and one stops kubectl after the cordon unless you pass --ignore-daemonsets.
Play the animation 5 steps, animated, best on a wide screen
Node Drain, step by step
-
The drain command PATCHes Node-1 with spec.unschedulable=true. The Scheduler stops placing new Pods on this Node unless they tolerate the node.kubernetes.io/unschedulable taint the way DaemonSet Pods do, and the status shows SchedulingDisabled. Already-running Pods stay put for now. Cordon is also exposed as a separate verb (kubectl cordon Node-1), drain just bundles it with the eviction loop.
Play from step 1 -
The drain command lists Pods on Node-1 via fieldSelector=spec.nodeName=Node-1 and buckets each one. A drain leaves running DaemonSet Pods in place and will not proceed without --ignore-daemonsets. Mirror Pods (the API record of static Pods) are skipped because Kubelet would recreate them. Running Pods with emptyDir volumes and running bare Pods with no owner abort the drain until the matching flag is passed. Two Deployment-backed Pods are left for the Eviction API.
Play from step 2 -
The drain command POSTs its evictions in parallel, and the one for web-1 reaches the API first. The API reads the matching PDB, whose status the disruption controller keeps at disruptionsAllowed=1. It decrements that to 0 under optimistic concurrency, marks web-1 with a DisruptionTarget condition and deletes it with its grace period, then answers 201 Created. The web-2 eviction lands just after, finds disruptionsAllowed=0 and gets 429 Too Many Requests. The ReplicaSet replaces web-1 elsewhere.
Play from step 3 -
After each 429 the drain command waits a fixed 5 seconds and POSTs the web-2 eviction again. While the web-1 replacement is still starting, currentHealthy=1 meets desiredHealthy=1, which minAvailable sets, so disruptionsAllowed stays 0. Once the replacement turns Ready elsewhere, the disruption controller counts currentHealthy=2 and raises disruptionsAllowed to 1. The next retry is granted with 201 Created, evicting web-2.
Play from step 4 -
Node-1 carries only the DaemonSet Pod now. Application traffic runs on the replacement web-1 and web-2 elsewhere. The Node is safe for kernel patch, reboot, or removal. To bring it back, kubectl uncordon Node-1 flips spec.unschedulable=false and the Scheduler can place new Pods on it again.
Play from step 5