Container OOMKill
A container blows past its memory limit, so what kills it and what gets recorded? The kernel cgroup enforces the limit, and once usage reaches the cap and reclaim cannot free enough, the cgroup out-of-memory killer SIGKILLs every process in that container at once under cgroup v2 by default, and it exits 137. The Kubelet only witnesses this. It records the OOMKilled reason in the container status, then restarts the container according to the Pod restart policy.
Play the animation 5 steps, animated, best on a wide screen
Container OOMKill, step by step
-
The workload grows, and memory.current keeps rising toward memory.max as the container allocates anonymous pages, page cache, and slab. The cgroup memory controller accounts every byte against the limit.
Play from step 1 -
Usage in memory.current reaches memory.max. The cgroup memory controller cannot reclaim enough (swap is disabled on most Kubernetes Nodes), so the kernel raises an out-of-memory event scoped to this one cgroup.
Play from step 2 -
Reclaim has failed at memory.max, so the kernel invokes the cgroup-scoped OOM killer. Under cgroup v2 the runtime set memory.oom.group on that cgroup at container start by default, so the kernel SIGKILLs every process in the container as one unit rather than the single worst offender. With the whole cgroup going, the oom_score_adj applied from the QoS class decides nothing here. It ranks containers when a wider scope runs out, the Pod cgroup or the whole Node.
Play from step 3 -
PLEG (Pod Lifecycle Event Generator) spots the dead container on its next relist of the container runtime. Kubelet PATCHes the container status to terminated with reason OOMKilled and exitCode 137 (128 + 9 for SIGKILL). Where that record goes once the container restarts is covered in the Container Restarts and lastState card.
Play from step 4 -
The restartPolicy is Always (the default), so Kubelet starts a fresh container inside the same Pod sandbox. The Pod IP and Linux namespaces are preserved and restartCount increments. Repeated OOMKills trip CrashLoopBackOff, and the backoff behind it is covered in the CrashLoopBackOff and Restart Backoff card.
Play from step 5