fsGroup and Volume Ownership
A freshly formatted volume is owned by root, so why can a non-root container not write to it? The securityContext.fsGroup field names a GID, and on a volume that supports it Kubelet chowns the whole tree to that group before the container starts, unless a CSI driver does it. The default policy Always redoes that walk on every start, which can add minutes on a volume with millions of files. OnRootMismatch skips it once the top directory matches.
Play the animation 6 steps, animated, best on a wide screen
fsGroup and Volume Ownership, step by step
-
Start the Pod with no fsGroup and it comes up, but the first write to the volume fails with permission denied. The disk is fine, the mount is fine. The ownership is simply wrong for a non-root process, and a process running as user 1000 cannot chown a tree that root owns.
Play from step 1 -
The fix is one field. Setting securityContext.fsGroup: 2000 asks Kubernetes to make the volume usable by group 2000, and every container in the Pod is added to that supplemental group. Kubelet reads this before it ever starts the container.
Play from step 2 -
Before the container starts, Kubelet walks the volume tree, chowns every entry to group 2000 and makes it group writable, setting the setgid bit on directories so new files inherit the group. The owner stays root. This is real work on real inodes, done once per mount. By default a CSI volume gets this only with an fsType, a ReadWriteOnce or ReadWriteOncePod access mode and a read-write mount. A CSI driver that advertises VOLUME_MOUNT_GROUP does the ownership itself instead, and then fsGroupChangePolicy has no effect at all.
Play from step 3 -
Now the container starts and its first write succeeds. Its process is in group 2000, the tree is group 2000, and the setgid bit keeps every new file in the same group. The permission problem is gone, paid for once at startup.
Play from step 4 -
There is a cost. The default policy, Always, walks the entire tree and sets the group and mode of every entry again on every Pod start, even when nothing changed. On a small volume that is nothing. On a volume with millions of files it can add minutes to each start, and the Pod sits waiting the whole time.
Play from step 5 -
The fsGroupChangePolicy: OnRootMismatch setting is the escape. Kubelet checks only the top-level directory: its group, its permission bits and its setgid bit. If all of them already match what fsGroup asks for, it assumes the tree was set on a previous start and skips the walk entirely. The next start pays no ownership walk, however many files sit below.
Play from step 6