Node Allocatable
Your Node reports 16Gi of memory, so why does the Scheduler refuse a 15Gi Pod? Capacity is what the machine has, and Allocatable is what survives after kubeReserved, systemReserved and the hard eviction threshold are carved out of it, which is the only number the Scheduler ever sums Pod requests against. Everything above Allocatable is spoken for, by the Kubelet, the runtime and the OS or by the eviction margin, so a half empty Node can still be full.
Play the animation 6 steps, animated, best on a wide screen
Node Allocatable, step by step
-
Node-1 has 16Gi of memory as its kernel counts it, a little under the RAM installed, and the Kubelet reports that total into status.capacity on the Node object. Capacity is the total the Node has, and it says nothing about who may use it. With nothing reserved beyond the Kubelet default eviction margin, 100Mi on a Linux Node, Pods can take nearly all of it, and competing with the daemons that keep the machine alive is exactly the problem the rest of this card solves.
Play from step 1 -
Two reservations come off the top, both Kubelet settings. The kubeReserved budget covers the Kubernetes daemons that are not Pods, such as the Kubelet and the container runtime, and here it is 1Gi. The systemReserved budget covers the OS daemons like sshd and udev, and should cover kernel memory too, which is not accounted to Pods, and here it is 512Mi. Both always come off Capacity, but neither becomes a real cgroup cap until you name it in enforceNodeAllocatable and set its reserved cgroup.
Play from step 2 -
The hard eviction threshold comes off as well. The evictionHard setting for memory.available, 512Mi on this Node, makes the Kubelet evict Pods once available memory drops below it, so the machine is much less likely to reach a kernel out-of-memory event. Even with no daemons on it at all, Pods could not use more than capacity minus this threshold, so the memory behind it counts as unavailable to Pods rather than as spare room.
Play from step 3 -
What survives is Allocatable. With no hugepages set aside on this Node, 16Gi minus 1Gi minus 512Mi minus 512Mi leaves 14Gi, and the Kubelet publishes that in status.allocatable beside status.capacity, which is why kubectl describe node prints the two blocks one under the other. Allocatable is the amount on this Node available to be consumed by ordinary Pods.
Play from step 4 -
The Scheduler reads status.allocatable, never status.capacity, and it does not over-subscribe it: the sum of Pod requests on a Node may not pass Allocatable. So Pod cache-0 asking for 15Gi is turned away by a Node advertising 16Gi, with a FailedScheduling event naming Insufficient memory, because the number it is measured against is 14Gi.
Play from step 5 -
Only requests are summed, plus any Pod overhead a RuntimeClass adds. Limits may add up far past Allocatable, which is what kubectl describe node means when it warns that total limits may be over 100 percent. Three Pods requesting 4Gi each fit inside 14Gi while their 8Gi limits total 24Gi. By default the Kubelet enforces Allocatable across Pods alone, and it enforces it by evicting once their real usage passes it.
Play from step 6