Image and Container GC
Your Nodes keep filling their own disks, so what does the Kubelet throw away first? It sweeps the image store once usage passes HighThresholdPercent, deleting images in the order they were last used and stopping the moment usage is back at LowThresholdPercent, and imageMaximumGCAge, once set, takes one nothing has touched for long enough whatever the disk reads. Dead containers are a separate budget, held to MinAge, MaxPerPodContainer and MaxContainers.
Play the animation 6 steps, animated, best on a wide screen
Image and Container GC, step by step
-
One Node fills its own disk from two directions, and the Kubelet counts the two stores separately. Container images pile up in the image store, dead containers pile up beside them, and each store has its own clock: unused images are collected every five minutes and unused containers every minute.
Play from step 1 -
The image manager inside the Kubelet does not measure the disk itself. It takes filesystem usage from cAdvisor and holds that reading against HighThresholdPercent, which is imageGCHighThresholdPercent in the Kubelet configuration and 85 percent by default. At 89 percent this Node is over the mark.
Play from step 2 -
Above the high mark the Kubelet deletes images in the order they were last used, oldest first, and stops once usage is back at LowThresholdPercent, imageGCLowThresholdPercent and 80 by default. With imageMaximumGCAge off, its default, the 41 day envoy:v1.31 goes, usage lands on 78, and fluentd:v1.17 survives a sweep it was next in line for.
Play from step 3 -
Turn imageMaximumGCAge on, 168h here, and every pass runs it first, before the disk is even read: an image unused for longer than that goes whatever the usage reads. So fluentd:v1.17 goes although 78 percent is already under the low mark. Image age is not tracked across a Kubelet restart, which starts the full wait again.
Play from step 4 -
Dead containers are a separate budget on three dials. MinAge is the floor below which nothing is collected, MaxPerPodContainer caps dead instances per container in a Pod, and MaxContainers caps the total and lowers that cap to fit. One container of Pod api-0 left three against a cap of two here, so its oldest goes, and the orphan goes once it passes MinAge.
Play from step 5 -
All of this is the Kubelet reclaiming Node resources before it has to touch a running Pod. Once a disk signal like imagefs.available crosses its own eviction threshold, the Kubelet first deletes every unused image past imageMinimumGCAge and starts failing Pods only if that is not enough, which is where the Node-pressure Eviction card picks the story up.
Play from step 6