Node Volume Attach Limits
Why does a Pod stay Pending on max volume count while every Node has spare CPU and memory? Because a Node can have a second, invisible capacity: how many volumes one CSI driver may have attached at once. The node plugin reports it, Kubelet writes it into CSINode, and the Scheduler rejects any Node the Pod would push past it. If the driver attaches, a slot frees when the detach completes, not when the Pod dies, turning a rollout near the ceiling into a race.
Play the animation 7 steps, animated, best on a wide screen
Node Volume Attach Limits, step by step
-
The ceiling is not a Kubernetes setting. The CSI node plugin reports it as max_volumes_per_node in NodeGetInfo, and the Kubelet writes it into CSINode as allocatable.count, rewriting it on a period if its CSIDriver sets one. Zero means no ceiling. Real drivers report from a handful to a hundred and twenty seven.
Play from step 1 -
Now the cluster fills. More Pods with claims are scheduled, more disks attach, and every Node walks up to its own ceiling: eight of eight on all three, twenty four of twenty four across the cluster. No alarm fires, because a Node sitting exactly at its ceiling is a healthy Node.
Play from step 2 -
Now Pod web-0 is created and asks for one volume of its own. Before the Scheduler scores any Node it filters out the ones that cannot take the Pod at all, and one filter exists for this ceiling: NodeVolumeLimits. It skips a Pod with no claim and no migrated in-tree volume.
Play from step 3 -
The filter reads allocatable.count for the driver from each CSINode and counts what the Node already holds of that driver: the volumes of its Pods, plus any VolumeAttachment none of them uses, each volume once. Eight plus the one web-0 needs is over eight, so all three are rejected before scoring.
Play from step 4 -
So web-0 stays Pending, and its event reads zero of three Nodes are available, three Nodes exceed max volume count. Every one of those Nodes has spare CPU and spare memory, which is what makes this hard to recognise: the cluster looks half empty and the Pod will not schedule.
Play from step 5 -
Here a detach clears it. For a driver that attaches, the VolumeAttachment holds the slot, not the Pod, so deleting a Pod frees nothing until that object is gone, usually seconds to tens of seconds later. One completes on Node-3, the count drops to seven, and web-0 is requeued and placed there.
Play from step 6 -
Every lever here is about the ceiling and none is about CPU. Fewer volumes per Pod is the cheapest: four claims of one driver take four of its slots, less any volume already in use on that Node. More Nodes buy more slots, and an instance type that reports a higher ceiling buys more per Node.
Play from step 7