MTU Overhead and PMTU Blackholes
Why does ping keep answering while a large transfer hangs on the same path? A VXLAN outer header spends 50 of the 1500 bytes a link carries, so the CNI sets the Pod eth0 to 1450 and the wrapped frame fits the Node NIC exactly. When one tunnel hop further along the path carries less and the ICMP that would report it is filtered, nothing on the Node ever learns, so small packets keep crossing while large ones disappear.
Play the animation 6 steps, animated, best on a wide screen
MTU Overhead and PMTU Blackholes, step by step
-
A link carries 1500 bytes in a frame and the overlay wrap is paid out of that number rather than added to it. A VXLAN header over IPv4 spends 50 of them, so 1450 are left for the frame a Pod puts inside. An IPIP wrap costs 20 and leaves 1480.
Play from step 1 -
That 1450 goes onto the interface that builds the frame. The CNI plugin sets the Pod eth0 MTU when it creates the interface, from its config or detected from the Node NIC, while the NIC keeps the full 1500. The cost lands one layer in, on the Pod, and the NIC number an operator reads does not show it.
Play from step 2 -
A ping is a tiny frame. The default 56 byte payload with its ICMP and IPv4 headers is an 84 byte datagram, and 134 on the wire once wrapped, nowhere near the limit. It crosses the underlay to Pod B and the echo comes home, which is why a cluster with an MTU problem still answers its health checks.
Play from step 3 -
The route to Node-2 crosses a VPN leg that carries only 1400 bytes, and nothing on the Node knows it. A full size frame is 1500 on the wire, 100 more than that hop can take, so where the tunnel sets the do not fragment bit the hop drops it and returns ICMP type 3 code 4 to the Node naming 1400.
Play from step 4 -
Where a firewall or a cloud security group filters ICMP, that reply never arrives and the Node never lowers anything. It keeps putting 1500 on a path that will not carry it. Pings and some interactive connections still work, while a bulk transfer fails on its first large packet and the connection times out.
Play from step 5 -
The fix that does not need ICMP is to clamp the TCP MSS. A rule on the path rewrites the value both ends announce in the handshake down to 1310, which is the 1400 on that hop less the outer header and the IPv4 and TCP headers, so no TCP segment is built too big. It does nothing for UDP or anything else with no MSS.
Play from step 6