Taints and Tolerations
Why does a Node refuse one Pod and happily run another? A taint on the Node is a key, an optional value and an effect, and what it does to a Pod with no toleration for that key and effect is decided by the effect alone. NoSchedule and PreferNoSchedule are gates on the way in, one a veto and one a score penalty, so neither disturbs a Pod already running. Only NoExecute evicts one, at once unless a matching toleration stops that or delays it by its tolerationSeconds.
Play the animation 6 steps, animated, best on a wide screen
Taints and Tolerations, step by step
-
A taint on the Node object carries a key, an optional value, and an effect. Node-1 takes dedicated=db:NoSchedule into spec.taints. Nothing on the Node stirs, because NoSchedule is a gate on the way in and says nothing about the Pods already bound here.
Play from step 1 -
A toleration is the matching field on the Pod. It matches when the keys are equal and the effects are equal, and the operator decides the value: Equal compares it, Exists ignores it. The toleration on Pod db-1 leaves the effect empty, which matches every effect on that key. Pod web-1 has none.
Play from step 2 -
Pod web-2 carries no toleration, so the TaintToleration filter plugin drops Node-1 before scoring and the Pod is recorded Unschedulable with a FailedScheduling event. Nothing reaches the Node, and Pod web-1 keeps running untouched, because it was bound before the taint arrived.
Play from step 3 -
The taint is rewritten with PreferNoSchedule. The same mismatch is now scored rather than filtered: the plugin marks Node-1 down among the Nodes that fit but does not remove it, so Pod web-2 can still land here. The control plane tries to avoid that and does not guarantee it.
Play from step 4 -
A second taint, dedicated=db:NoExecute, is added. This effect is the only one that reaches Pods already running: the taint-eviction-controller deletes Pod web-1 and Pod web-2, since neither tolerates it. Pod db-1 does, and with no tolerationSeconds set this taint never evicts it.
Play from step 5 -
A NoExecute toleration may carry tolerationSeconds, which holds the Pod that many seconds after the taint lands and then lets the eviction through. A Pod is given 300 for each built-in not-ready or unreachable taint it sets no toleration for, which is the few minutes on the Node Failure and Pod Recovery card.
Play from step 6