Node Restart and Reboot
A Node reboots, so what comes back and what does not? A Kubelet restart leaves the already running containers in place and a container runtime restart usually does too, while a reboot stops every one of them first and the Node reports NotReady until the Kubelet, the runtime and the network are ready. The Kubelet then recreates the containers of the Pods still bound to it, but a Pod deleted while the Node was away is replaced only if a controller owns it.
Play the animation 6 steps, animated, best on a wide screen
Node Restart and Reboot, step by step
-
Three different events get run together under one word and they are not the same thing. A Kubelet restart leaves the already running containers in place, and so does a container runtime restart in the configuration most of them run with. A reboot is the most disruptive of the three: it is both of the others at once.
Play from step 1 -
The machine reboots. Every container on the Node stops first, and the Kubelet and the container runtime go down with it. Graceful Node shutdown is off by default, so no Pod is marked Terminated and the objects stay bound here. The heartbeats pause, and after the 50s grace period the control plane marks the Node Unknown and taints it unreachable.
Play from step 2 -
The machine boots. The Kubelet starts, renews its Lease and reconciles the Node status, and the Node is reported NotReady until the Kubelet, the container runtime and the network are all ready. The control plane swaps the unreachable taint for node.kubernetes.io/not-ready.
Play from step 3 -
The Kubelet recreates the containers of the Pods still bound to this Node, and the Node goes Ready. The static Pod node-agent comes back with nothing else involved: it is managed by the Kubelet directly, without the API server observing it, so no controller has to place it again.
Play from step 4 -
If instead the Node stays down past the configured toleration period, the control plane deletes a Pod that does not tolerate the unreachable taint, and it is already marked for deletion by the time the machine returns. A ReplicaSet owns Pod web-7d4-x2k9p, so a replacement was created at once and the Scheduler may have put it on a different Node.
Play from step 5 -
Pod debug-shell went the same way, and nothing brings it back: a standalone Pod is not recreated after deletion. The static Pod is the opposite case, because deleting its mirror Pod through the API does not remove the static Pod the Kubelet runs. Run what has to survive a reboot under a controller.
Play from step 6