Force Deletion and Stuck Terminating Pods
Why does a Pod sit in Terminating forever when its Node goes silent? The Kubelet never acknowledges the delete, so the API keeps the object and an identity-bound controller like StatefulSet refuses to create a replacement. Force deletion drops it from ETCD at once, but a merely partitioned Node may still be running the original container. So a partition is waited out, and only a Node confirmed dead has its Node object deleted or its Pods force-deleted.
Play the animation 5 steps, animated, best on a wide screen
Force Deletion and Stuck Terminating Pods, step by step
-
Node-1 stops posting Kubelet heartbeats, from a kernel panic, a power loss or a network partition. After node-monitor-grace-period, 50s by default, the Ready condition goes to Unknown and Node-1 is marked NotReady. What goes with it is the acknowledgement channel, the one thing that keeps the stored object and the running container in step.
Play from step 1 -
A delete is issued for Pod A, named web-0, by hand here or automatically once its default 300s toleration for the unreachable taint runs out. The API stamps metadata.deletionTimestamp on the stored object, so kubectl prints Terminating. Normally the Kubelet stops the container, sets a terminal phase and asks the API to delete the object with grace 0. Here nothing reports back.
Play from step 2 -
Pod A is stuck in Terminating with no time limit, while status.phase stays Running. The StatefulSet creates no replacement, because the sticky identity web-0 and its RWO volume are still held by the object nobody removed. A leftover metadata.finalizers entry produces the same stuck Terminating, and there the fix is clearing the finalizer rather than reaching for force.
Play from step 3 -
Running kubectl delete pod web-0 --grace-period=0 --force tells the API to drop the object from ETCD at once, as it carries no finalizer, with no wait for any acknowledgement. The slot empties and the identity is free. Notice what did not happen: Node-1 is cut off and cannot see the change, so nothing stops the container, and the record now reads zero while it may still be running.
Play from step 4 -
The StatefulSet gives the freed name web-0 to Pod B on Node-2. If Node-1 was only partitioned, its Kubelet is alive and the original container keeps running and writing until the partition heals and that Kubelet kills it. One identity now has two live copies, a split brain that can lose data. So wait out a partition, and delete the Node object or force-delete only once the Node is confirmed dead.
Play from step 5