Detach on Node Failure
When a Node goes NotReady, why does its volume not move to a healthy Node? Nothing contends for it, the wait is on doubt: the old Pod cannot be confirmed dead, and detaching a disk it might still write to risks corruption. After the 300s toleration the Pod is marked for deletion but never confirmed gone, and only once its object is removed does a six minute force-detach clock start. The out-of-service taint skips both waits.
Play the animation 7 steps, animated, best on a wide screen
Detach on Node Failure, step by step
-
Node-1 stops answering. Its Kubelet goes silent and the Node is marked NotReady, but there is no word from Node-1 about whether the old Pod actually stopped. It might be dead. It might be a network blip with the Pod still writing.
Play from step 1 -
So Kubernetes will not detach the disk. Notice what it is not waiting on: no other Pod holds the volume and nothing is contending for it. It is waiting on doubt. Pull PV web off Node-1 while the old Pod might still be writing and its data can be corrupted. Holding on is the safe answer to a question that cannot be answered.
Play from step 2 -
The first clock starts. Node-1 takes the unreachable taint, and the old Pod tolerates it for 300 seconds by default before it is marked for deletion. A live Kubelet would finish that deletion and free the volume. Here nothing can, so the disk stays held, but the mark lets the ReplicaSet of the Deployment create a replacement on Node-2, stuck in ContainerCreating waiting for that disk.
Play from step 3 -
And there it stays. The old Pod object still exists and its container statuses still read running, so the controller keeps PV web wanted on Node-1 and never asks for a detach. No timeout runs. The Pod sits in Terminating with no time limit, and the replacement waits with it.
Play from step 4 -
The second clock starts only once the object is gone. Here kubectl force deletes it by hand, as PodGC would once the Node object is deleted. Roughly six minutes later, if Node-1 is still not Ready and the controller manager has not disabled it, the controller force-detaches the disk. Only now is it free.
Play from step 5 -
With PV web detached, it attaches to Node-2 and is mounted there, and the new Pod finally starts. Nothing in that sequence was slow. The outage was all waiting: the eviction toleration, however long the stuck Pod object lingered, and then six more minutes of deliberate doubt about a Node that could not be asked.
Play from step 6 -
If an operator has confirmed Node-1 is shut down, all that waiting is wasted downtime. The non-graceful shutdown escape hatch is the taint node.kubernetes.io/out-of-service with effect NoExecute: Pods without a matching toleration are force deleted and the volume is detached at once, with no six minute timer. A StatefulSet needs it even more: its replacement cannot be created at all while the old Pod keeps the name.
Play from step 7