Ephemeral Storage Limit Eviction
A Pod writes temp files, logs and scratch data to its Node, so why is it evicted while the Node disk is fine? The Kubelet periodically checks usage against limits.ephemeral-storage: a container counts its writable layer and logs, or only logs beside a separate image filesystem, and the Pod adds its disk emptyDir. Over the limit the whole Pod is evicted, not restarted, and stays Failed with reason Evicted while its controller, if any, makes a new one.
Play the animation 6 steps, animated, best on a wide screen
Ephemeral Storage Limit Eviction, step by step
-
Pod web-a has one container, app, with requests.ephemeral-storage 512Mi and limits.ephemeral-storage 1Gi. The request is what placement counts: web-a runs on Node-1 because 512Mi of its allocatable local storage was not yet requested. A request reserves room and caps nothing.
Play from step 1 -
The app writes three kinds of bytes to Node-1 local storage, all on one filesystem: temp files go to its writable layer, stdout to its log file, and /scratch is an emptyDir on disk. All three count for the Pod. The container counts its writable layer and log, or only the log on a Node with a separate image filesystem. The limit refuses no write.
Play from step 2 -
The Kubelet never sees a write happen. Every 10 seconds its eviction loop reads the usage it last measured, by periodic directory scans, or for an emptyDir by project quotas where their off by default beta gate is on, the Pod runs in a user namespace and the filesystem has project quotas. Pod web-a comes back at 1100Mi.
Play from step 3 -
Two sums meet the same 1Gi line. The container counts its writable layer and log, 700Mi, inside its own limit. The Pod adds the emptyDir: 1100Mi against the 1Gi sum of its container limits, and that one is over. The Kubelet checks any emptyDir sizeLimit first, then this Pod total, then each container.
Play from step 4 -
The Kubelet evicts the whole Pod, not one container. It kills web-a with a 1 second grace period, whatever its spec asks, though each container still gets the 2 second minimum stop. It sets phase Failed with reason Evicted and the message Pod ephemeral local storage usage exceeds the total limit of containers 1Gi. Node-1 was never short of disk.
Play from step 5 -
Nothing restarts web-a in place, unlike a container killed at its memory limit. It stays Failed, and its ReplicaSet creates a new Pod, web-b, which starts from zero and can land on Node-1 again. A Node that runs short of disk is a separate path, node-pressure eviction.
Play from step 6