Node Conditions

Up to five built-in conditions come back from kubectl describe node, so which of them keep a new Pod out and which one throws a running Pod off? Only the taints of Ready reach a Pod already there: False becomes the not-ready taint and Unknown becomes unreachable, and both carry NoExecute. MemoryPressure, DiskPressure, PIDPressure and NetworkUnavailable each become a NoSchedule taint, which shuts the door on new Pods and on its own leaves the running ones alone.

Play the animation 5 steps, animated, best on a wide screen

Node Conditions diagram: Node conditions and the taints they create

Node Conditions, step by step

  1. Under Conditions, kubectl describe node prints rows like these five, and they decide whether Pods may run here. Ready says the machine is healthy. MemoryPressure, DiskPressure and PIDPressure say it is short of memory, disk or process IDs, and NetworkUnavailable, which not every Node carries, says its network is not set up correctly. Ready is healthy at True, the other four at False.

    Play from step 1
  2. MemoryPressure, DiskPressure, PIDPressure and NetworkUnavailable all go True here at once, and the control plane writes a NoSchedule taint for each onto the Node, such as node.kubernetes.io/memory-pressure. The taints spare running Pods (what the Kubelet evicts under pressure is the Node-pressure Eviction card) and hold web-2 out. Being BestEffort is not why: by default a Burstable or Guaranteed Pod gets no memory-pressure toleration either.

    Play from step 2
  3. Ready False is the one condition whose taint reaches a Pod already there. The control plane writes node.kubernetes.io/not-ready with NoSchedule like the others, and again with NoExecute. Every Pod is given a 300 second toleration for that one unless it sets its own, so web-0 and web-1 go when that runs out, drawn here at once. The Node Failure and Pod Recovery card times it.

    Play from step 3
  4. Ready has a third value. When nothing has been heard from the Node for --node-monitor-grace-period, 50 seconds by default, Ready reads Unknown and the taint is node.kubernetes.io/unreachable instead. It carries NoExecute too, so False and Unknown differ in what went wrong and not in what it costs a Pod.

    Play from step 4
  5. The node-exporter Pod is still there. DaemonSet Pods are created with NoExecute tolerations for not-ready and unreachable, so they are never evicted for those two, and NoSchedule tolerations are added for memory, disk and PID pressure, plus network-unavailable on a hostNetwork Pod like this one.

    Play from step 5

Sources