Debugging Service Traffic

The Service exists and its name resolves, yet the app behind it never receives a single request from any client. Where along the path from the name to the Pod does the chain break, and how do you find it?

Play the animation 5 steps, animated, plays on a computer or a tablet

Debugging Service Traffic diagram: A Service call that never reaches the app

Debugging Service Traffic, step by step

  1. The client calls web on port 80 and the name resolves to the ClusterIP 10.96.0.20. The call reaches the app only while three links hold: the selector matches the Pod labels, the Pod is Ready so the slice serves it, and targetPort 9376 is the port the container listens on.

    Play from step 1
  2. The Pod carries app=web-v2 while the selector still asks for app=web. Nothing matches, so the Pod is not listed even though it is Ready, and the EndpointSlice holds no endpoints at all. The name still resolves and the client still sends, but with nothing behind the Service the call never reaches a Pod.

    Play from step 2
  3. The labels match again, but the Pod is not Ready, for example because its readiness probe fails. The slice still lists it, marked not ready and not serving, so the call never reaches a Pod. If the Service sets publishNotReadyAddresses, every endpoint is marked ready and the call gets through, though serving stays false.

    Play from step 3
  4. Now the Pod is Ready and the slice serves 10.244.2.7:9376, so the first two links hold. But the container listens on 8080 while targetPort still says 9376. A numeric targetPort goes into the endpoint as it is, so the call reaches the Pod on a port where nothing listens.

    Play from step 4
  5. Naming the port takes the number out of the Service. The targetPort becomes a port name, and the Pod declares containerPort 8080 under that name, so the endpoint resolves to 10.244.2.7:8080, where the container listens, and the call is served. If the port moves, only the Pod changes, and its declaration has to move with the process.

    Play from step 5

Sources