Multi-Attach Error
Why does a Pod hang in ContainerCreating with a Multi-Attach error, worded Waiting for detach since 1.36.4? An RWO disk attaches to one Node at a time, and by default RollingUpdate creates the replacement while the old Pod still holds the attachment. The controller will not write a second VolumeAttachment and the rollout will not remove the old Pod, so each waits on the other. Recreate breaks it by deleting first.
Play the animation 7 steps, animated, best on a wide screen
Multi-Attach Error, step by step
-
A rollout begins. At one replica, RollingUpdate with its default settings creates the replacement Pod before it removes the old one, exactly as designed, and the Scheduler places it on Node-2. Node-1 stays healthy throughout. A second Pod now exists on the other Node, and it wants the same volume.
Play from step 1 -
The attach and detach controller now has to attach the volume to Node-2, which for this CSI disk means writing a second VolumeAttachment. The request stops at the controller: PV web is ReadWriteOnce and still attached to Node-1, so it refuses before writing anything, and csi-2c9d stays a want.
Play from step 2 -
So the new Pod hangs in ContainerCreating. The controller records a FailedAttachVolume warning on it, Waiting for detach for volume web, naming Pod web-7f9c8-tp8vd as the one using it. Up to 1.36.3 it read Multi-Attach error. Kubelet cannot mount a disk not attached to its Node, so the container cannot start.
Play from step 3 -
Only the old attachment going away clears it. The controller will not delete csi-7f3a while the old Pod runs, and at one replica the default maxUnavailable rounds down to zero, so the rollout keeps that Pod until the new one is ready. Each side waits on the other, and progressDeadlineSeconds only reports the rollout as stalled.
Play from step 4 -
Nothing inside the rollout breaks the deadlock, and deleting the old Pod alone does not either: its ReplicaSet recreates it at once. So a hand from outside switches the strategy to Recreate, which scales the old ReplicaSet to zero. Kubelet on Node-1 unmounts, the controller deletes csi-7f3a, the disk detaches, and for a moment the volume belongs to nobody.
Play from step 5 -
With the volume free the controller writes csi-2c9d, the driver attaches the disk to Node-2, Kubelet mounts it, and the new Pod starts. None of that was the stall: it went on waiting for the old Pod to leave, not on any storage operation.
Play from step 6 -
So with default settings a Deployment on ReadWriteOnce storage stalls whenever the replacement lands on another Node: RollingUpdate creates the new Pod before deleting the old one. Set the strategy to Recreate, which deletes the old Pod first, the way a StatefulSet replaces an ordinal. The new Pod may still wait briefly for the detach.
Play from step 7