Container Restarts and lastState
A container crashed and restarted, so what killed it? The live state shows only the fresh Running instance, but Kubernetes keeps the previous termination beside it: lastState holds the exit code and reason, restartCount counts how often it has happened, and kubectl logs --previous reads the log stream of the instance that died. Together they are the whole post-mortem for a container you can no longer see, and only the last termination is kept.
Play the animation 6 steps, animated, best on a wide screen
Container Restarts and lastState, step by step
-
The process hits its memory limit and the kernel kills it. Kubelet spots the dead container and writes state as Terminated: a record carrying exitCode 137, reason OOMKilled, startedAt and finishedAt. It holds the live slot only until Kubelet turns to the restart, not until the new container runs.
Play from step 1 -
A container can leave a note. Whatever it writes to terminationMessagePath, default /dev/termination-log, becomes the message field of that Terminated record. The terminationMessagePolicy defaults to File, and FallbackToLogsOnError takes the log tail instead when the file is empty after a failed exit.
Play from step 2 -
Kubelet starts a fresh container in the same Pod and reports it. The record rolls over: state is Running again, the Terminated record moves into lastState, and what lastState held before is dropped. The restartCount becomes 2. On the Node the dead instance stays, container GC keeps one per container by default.
Play from step 3 -
This is where to look. The live state says Running, so the container is fine right now and explains nothing. Running kubectl describe pod prints both records, State and Last State, and Last State holds the answer: a Terminated record with exitCode 137 and reason OOMKilled, which is why the previous instance died.
Play from step 4 -
The exitCode names the cause. 0 is Completed, a clean exit. 1 is a generic application Error. Codes above 128 usually carry a signal: 137 is 128 plus 9 for SIGKILL, paired with reason OOMKilled when the kernel did it, and 143 is 128 plus 15 for SIGTERM. The number narrows the cause, the reason beside it names it.
Play from step 5 -
The record has no log. Running kubectl logs --previous asks for the log of the previous instance, and Kubelet reads it off the dead container on the Node. It works for one generation only: the next restart makes this instance the older one, container GC removes it, and its lastState record is overwritten too.
Play from step 6