Search Domains and ndots
Why can one lookup of an external name like api.example.com cost eight DNS queries from a Pod? With ndots:5 any name with fewer than five dots and no trailing dot is relative, so the resolver tries it with every search domain before trying it as written, and asks for A and AAAA each time. A Service in the same namespace answers on the first candidate, one in another namespace on the second, and a trailing dot skips the search list.
Play the animation 5 steps, animated, best on a wide screen
Search Domains and ndots, step by step
-
With the default ClusterFirst policy the Kubelet configures this resolv.conf: the kube-dns Service as nameserver, search domains built from the namespace and the cluster domain, and ndots:5. A name with fewer than 5 dots counts as relative, so the resolver tries it with every search domain first and as written last. Any search domains of the Node are appended after them.
Play from step 1 -
The name api has zero dots, so it is relative. The first candidate, api.default.svc.cluster.local, is a Service in the same namespace, so CoreDNS answers with its ClusterIP and the walk stops there. One name tried, and still two queries, because getaddrinfo asks for A and AAAA together.
Play from step 2 -
The name api.shop has one dot, still under 5. The first candidate, api.shop.default.svc.cluster.local, does not exist and comes back NXDOMAIN. The second, api.shop.svc.cluster.local, is that Service in the shop namespace and answers. This is what the search list is for: a name relative to the cluster resolves without its full suffix.
Play from step 3 -
The name api.example.com has two dots, so it is relative too, and no cluster suffix matches it. Three round trips end in NXDOMAIN before the name as written is asked and answered. Four names and eight queries for a single external lookup, paid again on every call, since the resolver inside the Pod keeps no cache. This is the real cost of ndots:5.
Play from step 4 -
A trailing dot makes a name absolute whatever ndots says, as in api.example.com., so the resolver skips every search domain and asks it once as written: one name, two queries. Fully qualifying hot external names, or lowering ndots through the Pod dnsConfig, is the usual fix for noisy cluster DNS.
Play from step 5