Pod Garbage Collection
Who deletes a Pod once it has finished? The object outlives the containers: a Succeeded or Failed Pod stays in the API until something removes it, usually PodGC in kube-controller-manager. It deletes terminated Pods, evicted first, then oldest, once they exceed terminated-pod-gc-threshold, 12500 by default, and three more rules ignore the count and take orphans, unscheduled terminating Pods, and terminating ones on a non-ready Node tainted out-of-service.
Play the animation 6 steps, animated, best on a wide screen
Pod Garbage Collection, step by step
-
A Pod whose containers have all exited for good is finished, and it is still an object. The phase reads Succeeded or Failed, nothing is running for it on any Node, and the API keeps the record: the object stays in the cluster until a human or a controller process explicitly removes it.
Play from step 1 -
PodGC is a control plane controller inside kube-controller-manager, and its first rule is a count. Terminated Pods are left alone while there are no more of them than terminated-pod-gc-threshold, which defaults to 12500. Over that, it deletes evicted ones first, then the oldest, until the count is back down to the threshold rather than under it. Set the flag to 0 or less and this rule is off.
Play from step 2 -
The other three rules never look at the count. PodGC watches Nodes as well as Pods, and a Pod matching any one of them goes whatever the number is. Because such a Pod is often still in a non-terminal phase, PodGC patches the phase to Failed before it deletes, so the record ends rather than the object simply disappearing.
Play from step 3 -
First rule of the three. Pod web-2 still names node-7 in spec.nodeName and that Node object has been deleted, so no Kubelet will ever report on it again. PodGC calls it an orphan and, in one status patch, sets the phase to Failed and adds a DisruptionTarget condition with reason DeletionByPodGC, then removes the object. No other PodGC rule adds that condition.
Play from step 4 -
Second rule, and it needs no Node at all. Pod web-3 carries a deletionTimestamp and never got a nodeName. The API removes an unscheduled Pod at once, so one that stays is held by a finalizer, and no Kubelet will ever give it a final phase. PodGC patches it to Failed and deletes it, and the object goes once the finalizer is cleared.
Play from step 5 -
Third rule, and this one waits for an operator. Pod web-4 is terminating on a Node that is not ready, and someone has confirmed the machine is shut down or powered off and tainted the Node node.kubernetes.io/out-of-service. PodGC takes that taint as the answer and force deletes the Pod, so the object stops waiting for an acknowledgement that is not coming.
Play from step 6