Eviction Rate and Unhealthy Zones
Half a rack goes NotReady at once, so does Kubernetes evict every Pod on all of them? The node controller limits its own rate: --node-eviction-rate 0.1/s taints one Node per 10 seconds, and once 3 or more Nodes and a --unhealthy-zone-threshold 0.55 share of a zone are unhealthy it drops to --secondary-node-eviction-rate 0.01, or to zero in a zone of --large-cluster-size-threshold 50 Nodes or fewer. Only while every zone is fully down does it taint nothing.
Play the animation 5 steps, animated, best on a wide screen
Eviction Rate and Unhealthy Zones, step by step
-
Every --node-monitor-period, 5 seconds by default, the node-lifecycle-controller re-reads the Ready condition of every Node and groups the answers by zone. Six of the sixty Nodes in zone us-east-1a are NotReady, a share of 0.10, and zone us-east-1b is untouched. Both shares are under --unhealthy-zone-threshold 0.55, so both zones count as healthy.
Play from step 1 -
A healthy zone is worked at --node-eviction-rate, 0.1 Nodes per second by default, which is one Node every 10 seconds. The controller takes Node-17 off the front of the queue for us-east-1a. Its Ready condition reads Unknown rather than False, so the key is node.kubernetes.io/unreachable:NoExecute, and on that taint the taint-eviction-controller, also in the controller-manager, starts evicting the Pods on it. Node-18 waits its turn.
Play from step 2 -
A rack switch fails and 36 of the 60 Nodes in us-east-1a stop answering. The share reaches 0.60, at or over the 0.55 threshold and past its 3 Node floor, so the zone is unhealthy and drops to --secondary-node-eviction-rate, 0.01 per second. A zone that loses every Node goes back to the normal rate instead, so the work can move out. The picture stands still because the next taint here is 100 seconds away.
Play from step 3 -
The outage spreads until not one Node in either zone answers, 60 of 60 in each. Only with every zone fully down does the node-lifecycle-controller read this as a problem between the control plane and the Nodes rather than as 120 dead machines. It drops every zone to a rate of zero and lifts the taint from Node-17, which goes back in the queue, so no Node is tainted while that holds.
Play from step 4 -
The partition clears. Most of us-east-1b answers again, 4 of 60 NotReady, and us-east-1a comes back to the 36 of 60 it had. No zone is fully down now, so each zone gets back its own rate, and its first taint waits one full interval: Node-42 in us-east-1b is tainted 10 seconds later, while Node-17, first in the us-east-1a queue, waits 100. The Pods on the Nodes that stayed unreachable are evicted after all.
Play from step 5