Node Failure and Pod Recovery
A Node goes silent, so how long until its Pods come back elsewhere? The cluster notices when the heartbeat Lease stops renewing, flips the Node to an unknown state, and taints it so that the Pods are eventually evicted and rescheduled somewhere healthy. The grace period plus the default toleration mean recovery takes a few minutes by design, trading raw speed for not overreacting to a network blip.
Play the animation 6 steps, animated, best on a wide screen
Node Failure and Pod Recovery, step by step
-
Kubelet on Node-1 proves liveness with two heartbeats. It renews its Lease in kube-node-lease every 10s and PATCHes Node.status when it changes or else every 5 min. The node-lifecycle-controller counts either as a sign of life, but the Lease renewal is the one that lands well inside its 50s grace period.
Play from step 1 -
The Kubelet on Node-1 stops renewing (kernel panic, network partition, or Kubelet crash). The Lease grows stale, but Pods on the Node keep running for now.
Play from step 2 -
After --node-monitor-grace-period (default 50s), the node-lifecycle-controller flips Ready from True to Unknown: it cannot tell whether Node-1 died or is just unreachable. Pods are still on the Node, and eviction has not started.
Play from step 3 -
The node-lifecycle-controller adds the taint node.kubernetes.io/unreachable:NoExecute. Kubernetes had already given this Pod a 300s toleration for it, which it does for any Pod that does not set one itself. DaemonSet Pods set theirs with no tolerationSeconds, so this never evicts them. The 300s now ticks down.
Play from step 4 -
Toleration expires. The taint-eviction-controller marks the Pod with a DisruptionTarget condition, then deletes it with a plain DELETE that bypasses PodDisruptionBudgets (unlike kubectl drain, which uses the PDB-aware Eviction API). The Pod gets a deletionTimestamp and sits in Terminating: the unreachable Node-1 still holds the orphaned container, and the entry clears only when the Kubelet answers, the Node object is deleted, or someone forces it.
Play from step 5 -
The owning controller (Deployment via its ReplicaSet) sees the missing replica and creates a replacement Pod. Scheduler picks the healthy Node-2 and Kubelet there starts it. End-to-end recovery takes about 50s plus 300s by default, the grace period plus the toleration.
Play from step 6