NodeLocal DNSCache
Why does cluster DNS stall under load, and what does a cache on every Node change? Without one, each lookup from a ClusterFirst Pod is a UDP query that kube-proxy rules DNAT to a CoreDNS Pod, often on another Node, with a conntrack entry per lookup. In iptables mode NodeLocal DNSCache answers that ClusterIP on the Node with no DNAT or conntrack, sends cluster misses to CoreDNS over TCP and other names to Node resolvers, and caches answers for up to 30 seconds.
Play the animation 5 steps, animated, best on a wide screen
NodeLocal DNSCache, step by step
-
Without a local cache, each lookup from a ClusterFirst Pod is a UDP query to the kube-dns ClusterIP 10.96.0.10. The kube-proxy rules on Node-1 DNAT it to a CoreDNS Pod, here on Node-2, and the flow opens a conntrack entry that must time out. When its A and AAAA queries race on that entry, one can be dropped, and the resolver waits out its five second timeout.
Play from step 1 -
A DaemonSet runs node-local-dns, a caching CoreDNS, on the host network of every Node. In iptables mode it binds a link-local address, here 169.254.20.10, and the kube-dns ClusterIP with NOTRACK rules, so the same query stops at the agent: no DNAT, no conntrack entry, no change to Pods. IPVS mode, deprecated, binds only the first and needs the Kubelet clusterDNS changed.
Play from step 2 -
The cache is empty, so the agent forwards the cluster name to CoreDNS. It already owns 10.96.0.10, so it dials a second Service, kube-dns-upstream, over TCP. That leg is DNAT-ed and tracked, but a TCP entry is removed when the connection closes. The answer is cached, then returned to the Pod.
Play from step 3 -
The next lookup for that name is answered from the cache and never leaves Node-1. A positive answer is kept for its record TTL, held between 5 and 30 seconds, and a negative one for 5. Once it expires, the next lookup goes to CoreDNS again.
Play from step 4 -
By default only the cluster domain and the reverse zones go to CoreDNS. A name such as example.com matches the catch-all zone of the agent, which forwards to the resolvers in the resolv.conf of the Node, and the answer is still cached on Node-1. Of that lookup only the ndots:5 search-list tries, which are cluster names, reach CoreDNS.
Play from step 5