Graceful Pod Shutdown
Why does a deleted Pod still take traffic for a moment, and why does it not die on the spot? One DELETE stamps deletionTimestamp on the stored object and two tracks start: the control plane marks the endpoint terminating so kube-proxy stops choosing it while others stay ready, and the Kubelet runs preStop, has the runtime send the stop signal and spends the same 30s window. Neither waits for the other, preStop lines them up, and SIGKILL at 0s is the last resort.
Play the animation 6 steps, animated, best on a wide screen
Graceful Pod Shutdown, step by step
-
A DELETE for Pod web-0 reaches the API, which removes nothing: it stamps deletionTimestamp on the stored object and records the 30s grace period beside it. Two watchers see that one write and start at once, the EndpointSlice controller and the Kubelet on Node-1. From here on kubectl prints Terminating while status.phase stays Running, and new connections are still reaching the Pod.
Play from step 1 -
The EndpointSlice controller never waits for the Node. It marks the endpoint for 10.244.1.7 terminating, ready false and serving still true, and kube-proxy on every Node picks that up and stops choosing the address for new connections while the Service has other ready endpoints. Propagation takes real time, and established connections keep flowing on their conntrack entries.
Play from step 2 -
The Node side runs the preStop hook first, before any signal is sent. A short sleep here is the usual way to line the two tracks up: it holds PID 1 alive while kube-proxy finishes deregistering the endpoint, and in-flight requests complete during the pause. The five seconds the hook takes come out of the same 30s window.
Play from step 3 -
Once preStop returns, the Kubelet asks the runtime to deliver the stop signal to PID 1, SIGTERM unless the image sets a different STOPSIGNAL. A well-behaved app stops accepting work, drains what it still holds, closes its connections and pools and exits on its own, usually well inside the window. The timer keeps running through all of it.
Play from step 4 -
If a container is still running when the window reaches 0, the runtime sends SIGKILL to every process still alive in any container of the Pod, not just to PID 1. Whatever the app was still doing is lost, which is why the window has to cover the preStop pause plus the real drain time, and why a hook that outlives the window is cut short after a 2s extension, not waited for.
Play from step 5 -
Either way, once every container has stopped the Kubelet sets the terminal phase, Failed or Succeeded, cleans up on the Node and asks the API to delete the object with grace 0. The API removes it from ETCD and kubectl no longer finds web-0. The EndpointSlice controller drops the endpoint from the slice, as it does for any terminal Pod, and kube-proxy forgets the address.
Play from step 6