Static Pods and Mirror Pods
If the API server runs as a Pod, who starts it before the API server exists? The Kubelet watches a manifest directory on the Node and runs every manifest there except dot files, with no Scheduler or controller involved, which is how kubeadm brings up the control plane on a fresh machine. For each it also creates a read-only mirror Pod in the API so kubectl sees it, and deleting that mirror changes nothing because the file on disk stays the source of truth.
Play the animation 6 steps, animated, best on a wide screen
Static Pods and Mirror Pods, step by step
-
A Pod manifest appears in the directory the Kubelet watches on Node-1, named by staticPodPath in the KubeletConfiguration and conventionally /etc/kubernetes/manifests. The Kubelet rescans it and reads every file whose name does not start with a dot.
Play from step 1 -
The Kubelet starts the container itself. No Scheduler placed this Pod and no controller owns it, so the Kubelet supervises it directly and restarts it when it fails. That is how a kubeadm control plane comes up: the API server, the controller-manager and the Scheduler run as static Pods, and so does ETCD in the default stacked topology.
Play from step 2 -
The Kubelet also creates a mirror Pod in the API for it, so kubectl get pods lists it like any other Pod. The name takes the Node name as a suffix, the kubernetes.io/config.mirror annotation marks it, and the labels on the file are copied across so selectors match it.
Play from step 3 -
Deleting the mirror Pod with kubectl removes the API object and nothing else. The container on Node-1 keeps running, because the file on disk is what the Kubelet reads, and the Kubelet puts the mirror back on its next sync loop pass. Nothing done to the object reaches the container.
Play from step 4 -
To change a static Pod you change its file. When the Kubelet picks up the edit it stops the old Pod, then replaces the mirror Pod to match and starts a new Pod from the new spec. Moving the file out of the directory removes the Pod. The spec cannot refer to other API objects such as a ConfigMap, a Secret or a ServiceAccount, so everything it needs comes off the file or the Node filesystem.
Play from step 5 -
A drain evicts or deletes the Pods on Node-1 and skips every mirror Pod, because removing one through the API would stop nothing. DaemonSet Pods are left alone too, and the Node Drain card covers the rest of that loop. So a static Pod rides out a drain and a kubeadm control plane keeps serving while its Node is cordoned.
Play from step 6