CPU Throttling and CFS Quota
A container over its memory limit is killed, so why is one over its CPU limit still alive? A CPU limit becomes a CFS quota, a budget of run time the cgroup may spend inside every 100ms period, and once it is spent the kernel stops scheduling those threads until the next period opens. The quota kills nothing, the latency just grows, which is why throttling shows up in container_cpu_cfs_throttled_seconds_total and not in kubectl describe.
Play the animation 5 steps, animated, best on a wide screen
CPU Throttling and CFS Quota, step by step
-
The Kubelet sends the CPU request down the CRI and it lands as a cgroup v2 cpu.weight. A weight is not a reservation. It only decides how the runnable cgroups on this Node divide the CPUs when they all want to run at once, so on a quiet Node this container may run past its 250m, and what caps it is the limit rather than the request.
Play from step 1 -
The CPU limit becomes a quota. On cgroup v2 that is one line, cpu.max, carrying the quota and the period together, so limits.cpu 500m against the default 100ms period is 50000 100000. That is 50ms of run time this cgroup may spend inside every 100ms period, and the period repeats for as long as the container lives.
Play from step 2 -
The container runs, and every microsecond of CPU time it burns is charged against the 50ms budget. One busy thread on one CPU empties it 50ms into the period. Nothing is wrong with the code, it has simply reached the ceiling that the limit bought.
Play from step 3 -
With the budget gone the kernel takes the cgroup off the run queues until the period timer refills it, so for 50ms of every 100ms the container is runnable and not running. Threads share one budget, so four busy threads on four CPUs would empty it 12.5ms in and stall for the remaining 87.5ms.
Play from step 4 -
Nothing died. The Pod stays Running with restartCount 0, and kubectl describe shows no event and no condition: no Kubernetes object carries throttling. The kernel counts the stall in cpu.stat and, with kernel PSI on, in cpu.pressure, and cAdvisor inside the Kubelet exports both. Its container_cpu_cfs_throttled_seconds_total only climbs: 100 periods in, it reads 5 seconds. The quota kills nothing, but the latency can fail a liveness probe, which restarts the container.
Play from step 5