Pod cgroup Hierarchy
Where do the CPU and memory limits on a Pod actually land? The Kubelet builds one cgroup v2 tree on every Node: kubepods.slice holds all end-user Pods, a QoS slice groups the Burstable and the BestEffort ones, and every Pod gets a slice of its own. The container runtime creates the last level, one leaf per container, and writes cpu.max, memory.max and cpu.weight there, while the Pod slice above it carries the summed limits when every container sets them.
Play the animation 6 steps, animated, best on a wide screen
Pod cgroup Hierarchy, step by step
-
On a cgroup v2 Node the whole tree is one unified hierarchy mounted at /sys/fs/cgroup, and stat -fc %T on that path answers cgroup2fs. The Kubelet and the container runtime both write into it, so they have to agree on one cgroup driver, and with cgroup v2 that is the systemd driver on both.
Play from step 1 -
Under that root the Kubelet creates kubepods.slice and parents every end-user Pod below it, which is what cgroupsPerQOS does and it is on by default. The default enforceNodeAllocatable setting then makes Allocatable the ceiling over that whole subtree, enforced by evicting once the Pods together pass it.
Play from step 2 -
One level down, kubepods.slice has three kinds of child. Burstable Pods go under kubepods-burstable.slice and BestEffort Pods under kubepods-besteffort.slice, so a value written on one QoS cgroup reaches the whole tier below it. A Guaranteed Pod gets no QoS slice: its own slice hangs straight off kubepods.slice.
Play from step 3 -
When Pod web-0 starts, the Kubelet gives it a slice of its own inside the tier its QoS class picked, named for the Pod uid. That slice is the parent of every container in the Pod, so the Pod is accounted in one place whatever its containers do inside it.
Play from step 4 -
The container runtime creates the last level, one leaf cgroup per container, and writes the values into it: cpu.max from limits.cpu, memory.max from limits.memory in bytes, cpu.weight from requests.cpu. On runc 1.3.2 and crun 1.23 or later, 250m maps to weight 35, on older ones to 10. The no-internal-process rule keeps processes on the leaves, so this is the cgroup the app runs in.
Play from step 5 -
The tree is where the numbers sit, not what acts on them. Two cards pick them up from here: CPU Throttling and CFS Quota takes cpu.max as the budget a container may spend inside each period, and Container OOMKill takes memory.max as the cap that gets a container killed. Read those two next.
Play from step 6