Graceful Node Shutdown
When a Node is told to power off, can its Pods still shut down cleanly first? With graceful Node shutdown enabled, the Kubelet catches the shutdown signal from systemd while already holding an inhibitor lock that delays the power-off, and it terminates Pods in priority order, spending the front of that one budget on ordinary workloads and the tail on the critical ones. Only then does it release the lock and let the operating system finish powering down.
Play the animation 5 steps, animated, best on a wide screen
Graceful Node Shutdown, step by step
-
The Node is about to shut down (poweroff, reboot, or halt), and systemd emits PrepareForShutdown over D-Bus. Kubelet catches the signal via its logind subscription. Its delay-type inhibitor lock makes systemd pause the actual shutdown, so Kubelet can enter shutdown mode rather than let the OS kill processes outright.
Play from step 1 -
Kubelet sets the Node NotReady with reason KubeletNotReady and the message node is shutting down. The Scheduler stops placing anything here, and the Kubelet admission handler rejects even Pods tolerating the not-ready taint. Pods are bucketed by priority: at or above 2,000,000,000 is critical, the rest non-critical.
Play from step 2 -
As it sets NotReady, Kubelet also kills every non-critical Pod in parallel, preStop hook first if any, then SIGTERM. The window is the 60s budget less the 20s kept for critical Pods, so 40s, or the Pod terminationGracePeriodSeconds if shorter. Each ends with the status reason Terminated.
Play from step 3 -
Once the non-critical Pods are killed and their volumes unmounted, or the 40s runs out, Kubelet kills the system-critical Pods the same way, in parallel. Their window is shutdownGracePeriodCriticalPods (20s here), also capped by terminationGracePeriodSeconds. DaemonSet infra like CNI or kube-proxy usually sits here.
Play from step 4 -
The critical Pods are done or their 20s ran out. Kubelet releases the inhibitor lock, and systemd resumes the shutdown. When the Node Lease in kube-node-lease expires, the node-lifecycle-controller turns Ready from False to Unknown and swaps the not-ready taint for unreachable, NoSchedule and NoExecute alike.
Play from step 5