Graceful Node Shutdown

A Node is told to power off while its Pods are still serving requests, and the system would normally just kill them all. Can they shut down cleanly first, in what order, and how long does each of them get?

Play the animation 5 steps, animated, plays on a computer or a tablet

Graceful Node Shutdown diagram: Graceful Node shutdown, where the Kubelet holds a delay inhibitor lock, systemd signals

Graceful Node Shutdown, step by step

  1. The Node is about to shut down (poweroff, reboot, or halt), and systemd emits PrepareForShutdown over D-Bus. Kubelet catches the signal via its logind subscription. Its delay-type inhibitor lock makes systemd pause the actual shutdown, so Kubelet can enter shutdown mode rather than let the OS kill processes outright.

    Play from step 1
  2. Kubelet sets the Node NotReady with reason KubeletNotReady and the message node is shutting down. The Scheduler stops placing anything here, and the Kubelet admission handler rejects even Pods tolerating the not-ready taint. Pods are bucketed by priority: at or above 2,000,000,000 is critical, the rest non-critical.

    Play from step 2
  3. As it sets NotReady, Kubelet also kills every non-critical Pod in parallel, preStop hook first if any, then SIGTERM. The window is the 60s budget less the 20s kept for critical Pods, so 40s, or the Pod terminationGracePeriodSeconds if shorter. Each ends with the status reason Terminated.

    Play from step 3
  4. Once the non-critical Pods are killed and their volumes unmounted, or the 40s runs out, Kubelet kills the system-critical Pods the same way, in parallel. Their window is shutdownGracePeriodCriticalPods (20s here), also capped by terminationGracePeriodSeconds. DaemonSet infra like CNI or kube-proxy usually sits here.

    Play from step 4
  5. The critical Pods are done or their 20s ran out. Kubelet releases the inhibitor lock, and systemd resumes the shutdown. After the monitor grace period with no Lease renewal, the node-lifecycle-controller turns Ready from False to Unknown and swaps the not-ready taint for unreachable, NoSchedule and NoExecute alike.

    Play from step 5

Sources