CrashLoopBackOff and Restart Backoff
Each time the Kubelet restarts a broken container, the pause before the next try grows longer, until it hits a ceiling. Why the slowdown, what does kubectl show during the wait, and what finally resets the pause?
Play the animation 5 steps, animated, plays on a computer or a tablet
CrashLoopBackOff and Restart Backoff, step by step
-
The container process exits with a non-zero code, and Kubelet notices it. With restartPolicy Always, Kubelet restarts the container immediately the first time and arms a 10s base delay for the next one. Once the new container starts, restartCount becomes 1.
Play from step 1 -
The fresh container crashes again almost immediately. This second restart is the one that waits 10s, and it arms a doubled 20s for the next one. While Kubelet waits before the third restart, the container sits in Waiting with reason CrashLoopBackOff, which shows in kubectl get pods.
Play from step 2 -
The container keeps crashing, and the backoff delay doubles with each restart, climbing to 40s, then 80s, then 160s. The restartCount grows on every attempt. The backoff is kept per container, and it stops a hot-looping process from saturating the Node.
Play from step 3 -
The next doubling would exceed 300s, so the delay is clamped at the 300s ceiling, a per-Node default since 1.35, and stays there. Kubelet now retries the container at most once every 5 minutes for as long as it keeps failing. The restartCount continues to climb at this slow cadence.
Play from step 4 -
The bug is fixed and the new container runs stably. After 10 healthy minutes Kubelet resets the backoff. A new crash then counts as a first one: the container is restarted at once, and the next wait is 10s, not 300s. The container returns to Running and the CrashLoopBackOff reason clears.
Play from step 5