Ephemeral Debug Containers
A container runs an image with no shell and no tools, so there is nothing to exec into, and restarting the Pod is not an option either. How do you look inside while it keeps running, and see its processes and files?
Play the animation 6 steps, animated, plays on a computer or a tablet
Ephemeral Debug Containers, step by step
-
The app container runs a distroless image, which ships no shell and no debugging tools. The kubectl exec command goes through the pods/exec subresource and asks the container to start sh, but there is nothing to start. So the first door is closed by the image itself, and no shell exec into app can open it.
Play from step 1 -
Adding a container is closed too. The kubectl edit command writes the Pod spec back through the Pod resource itself, and a container cannot be appended to spec.containers on a live Pod: through that door a Pod update may change nothing in the list but the images, so the API server rejects it. The spec.ephemeralContainers list cannot be set that way either.
Play from step 2 -
The kubectl debug command builds an EphemeralContainer that targets app, names it debugger- plus a random suffix when you do not (debugger-8xzrl here), and PATCHes it onto the pods/ephemeralcontainers subresource. Only that handler writes the list, so kubectl edit cannot reach it. The API server appends the entry, which can never be changed or removed afterwards.
Play from step 3 -
The object going through that door takes the same container spec as a real container, with many fields refused. It may not have ports, so ports, livenessProbe and readinessProbe are disallowed. Resources are disallowed too, because it spends spare capacity already allocated to the Pod, and restartPolicy cannot be set: it is never restarted once it exits.
Play from step 4 -
The Kubelet on the Node sees the new entry and starts the container inside the already running Pod, in the namespaces of the container targetContainerName names, PID among them. The Pod is not replaced and app is not restarted, so the restartCount of app stays at 0, and what the debug container does is reported under status.ephemeralContainerStatuses.
Play from step 5 -
The debug container shares the PID namespace of app, so ps in it lists the app process, and the /proc link of that process reaches the app filesystem, which is how a distroless image gets read. With its default general profile kubectl debug adds SYS_PTRACE to the debug container, and the container runtime must implement namespace targeting for any of it to work.
Play from step 6