Static Pods and Mirror Pods

In many clusters the API server itself runs as a Pod, yet deleting that Pod with kubectl stops nothing at all, and it soon shows up again. Who started it before any API existed, and why does it simply ignore you?

Play the animation 6 steps, animated, plays on a computer or a tablet

Static Pods and Mirror Pods diagram: The Kubelet runs a static Pod from a manifest file on the Node with no Scheduler and no

Static Pods and Mirror Pods, step by step

  1. A Pod manifest appears in the directory the Kubelet watches on Node-1, named by staticPodPath in the KubeletConfiguration and conventionally /etc/kubernetes/manifests. The Kubelet rescans it and reads every file whose name does not start with a dot.

    Play from step 1
  2. The Kubelet starts the container itself. No Scheduler placed this Pod and no controller owns it, so the Kubelet supervises it directly and restarts it when it fails. That is how a kubeadm control plane comes up: the API server, the controller-manager and the Scheduler run as static Pods, and so does ETCD in the default stacked topology.

    Play from step 2
  3. The Kubelet also creates a mirror Pod in the API for it, so kubectl get pods lists it like any other Pod. The name takes the Node name as a suffix, the kubernetes.io/config.mirror annotation marks it, and the labels on the file are copied across so selectors match it.

    Play from step 3
  4. Deleting the mirror Pod with kubectl removes only the API object, and nothing done to the object reaches the container. The container on Node-1 keeps running because the Kubelet reads the file on disk. On its next sync loop pass the Kubelet puts the mirror back.

    Play from step 4
  5. To change a static Pod you change its file. When the Kubelet picks up the edit it stops the old Pod, then replaces the mirror Pod to match and starts a new Pod from the new spec. Moving the file out of the directory removes the Pod. The spec cannot refer to other API objects such as a ConfigMap, a Secret or a ServiceAccount, so what those objects would hold comes off the file or the Node filesystem.

    Play from step 5
  6. A drain evicts or deletes the Pods on Node-1 and skips every mirror Pod, because removing one through the API would stop nothing. DaemonSet Pods are left alone too, and the Node Drain card covers the rest of that loop. So a static Pod rides out a drain and a kubeadm control plane keeps serving while its Node is cordoned.

    Play from step 6

Sources