fsGroup and Volume Ownership

A non-root container can hit permission denied on its volume, and the usual fix, fsGroup in the Pod, can stretch each Pod start on a big volume to minutes. Who changes the group of the volume files, and how often?

Play the animation 6 steps, animated, plays on a computer or a tablet

fsGroup and Volume Ownership, Kubernetes diagram

fsGroup and Volume Ownership, step by step

  1. A Pod started with no fsGroup comes up, but the first write to the volume fails with permission denied. The disk and the mount are fine, but the ownership is wrong for a non-root process. And a process running as user 1000 cannot chown a tree that root owns.

    Play from step 1
  2. The fix is one field. Setting securityContext.fsGroup: 2000 asks Kubernetes to make the volume usable by group 2000, and every container in the Pod is added to that supplemental group. Kubelet reads this before it ever starts the container.

    Play from step 2
  3. Before the container starts, Kubelet walks the volume tree, chowns every entry to group 2000 and makes it group writable, setting the setgid bit on directories so new files inherit the group. The owner stays root. This is real work on real inodes, done once per mount. By default a CSI volume gets this only with an fsType, a ReadWriteOnce or ReadWriteOncePod access mode and a read-write mount. A CSI driver that advertises VOLUME_MOUNT_GROUP does the ownership itself instead, and then fsGroupChangePolicy has no effect at all.

    Play from step 3
  4. Now the container starts and its first write succeeds. Its process is in group 2000, the tree is group 2000, and the setgid bit keeps every new file in the same group. The permission problem is gone, paid for once at startup.

    Play from step 4
  5. But the default policy, Always, has a cost. On every Pod start Always walks the entire tree and resets the group and mode of every entry, even when nothing changed. On a small volume that is nothing. On a volume with millions of files it can add minutes to each start, and the Pod sits waiting the whole time.

    Play from step 5
  6. The fsGroupChangePolicy: OnRootMismatch setting avoids that. Kubelet checks only the top-level directory, that is its group, its permission bits and its setgid bit. If all of them already match what fsGroup asks for, Kubelet assumes the tree was set on a previous start and skips the whole walk, however many files sit below.

    Play from step 6

Sources