Kubernetes Services: a virtual IP with nothing behind it
A pod's IP address is a bad thing to depend on. Pods get rescheduled, replaced on every deploy, and scaled from three to thirty and back — and each new pod gets a new address. Anything holding a pod IP is holding a value that expires without notice.
A Service is the fix: one address and one name that stay constant while the pods behind them are replaced. The mechanism is worth understanding precisely, because it's unlike anything else in this course — the address a Service answers on belongs to no interface anywhere in the cluster.
The flat network assumption
Kubernetes starts from a rule that sounds unremarkable and has large consequences: every pod gets its own IP address, and every pod can reach every other pod's IP directly, without NAT, on any node.
NAME READY STATUS IP NODE
api-7d4b9c8f5-2xk4p 1/1 Running 10.244.1.7 node-2
api-7d4b9c8f5-9mnzt 1/1 Running 10.244.2.3 node-3
redis-0 1/1 Running 10.244.1.9 node-2
Three pods on two nodes, all in 10.244.0.0/16, all mutually reachable. Whoever implements that — the CNI plugin, a component you choose per cluster — does it either by tunnelling (VXLAN, like the overlay driver) or by programming real routes into the underlying network. Which one your cluster uses determines its MTU and how it interacts with cloud routing, but the pod-level contract is the same either way.
Note what this contract removes. There's no NAT between pods, so a pod sees the real source IP of another pod, and every pod's port space is its own — a hundred pods can each bind port 8080. Also note what it doesn't give you: those addresses are still ephemeral.
The ClusterIP that isn't on any machine
10.96.184.22 is stable for the Service's whole lifetime. Now try to find the machine that owns it:
PING 10.96.184.22 (10.96.184.22): 56 data bytes
--- 10.96.184.22 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
No reply. Nothing answers ICMP, nothing answers ARP, no interface has this address. And yet:
The TCP connection works. A ClusterIP is not an address you send packets to — it's a pattern the kernel rewrites. When a packet leaves a pod destined for 10.96.184.22:80, a rule on that node's stack changes the destination to one of the real pod addresses before the packet is routed anywhere. Nothing ever receives traffic as 10.96.184.22.
That rewriting is DNAT, the same mechanism as Docker's port publishing, and the rules are written by kube-proxy on every node. In iptables mode they look roughly like this:
Chain KUBE-SERVICES (2 references)
target prot opt source destination
KUBE-SVC-XKNZ3G tcp -- 0.0.0.0/0 10.96.184.22 tcp dpt:80
Follow KUBE-SVC-XKNZ3G and you find one branch per pod, selected probabilistically — which is where a Service's load balancing actually happens. It's random selection per connection, not round robin, and it has no idea how loaded any pod is.
Two consequences that regularly confuse people, both explained by this design:
- A ClusterIP doesn't answer
ping. ICMP isn't matched by the DNAT rules, which target TCP or UDP on a specific port. A Service failing to respond topingis normal and proves nothing. - The connection is only load-balanced at connect time. Once established, a TCP connection stays pinned to the pod it was rewritten to. A client holding a long-lived connection — a database pool, a gRPC channel, an HTTP/2 connection — keeps hitting one pod forever, and scaling up doesn't redistribute existing traffic. This is a real production surprise: you add pods, and load doesn't move.
What decides which pods are behind it
Not the Service. A selector matches labels on pods, and the set of matching, ready pods becomes the Service's endpoints:
Two endpoints, matching the two pods. This is the first thing to check when a Service isn't working, because the failure mode is silent:
No endpoints means the DNAT rules have nothing to rewrite to, and connections to the ClusterIP get refused or hang. The Service object still exists, kubectl get svc looks perfectly healthy, and nothing reports an error. Two causes account for nearly all of it: the selector doesn't match the pods' labels (a typo, or a label changed in the deployment but not the service), or the pods are running but failing their readiness probe, which removes them from the endpoint list by design.
kubectl get endpointslicebefore anything else. An empty endpoint list turns a vague "the service is down" into a specific label or readiness problem in one command.
The other classic mismatch is targetPort. port is what the Service listens on; targetPort is the port on the pod. Set targetPort: 8080 when the container listens on 3000 and you get endpoints that exist and connections that are refused — a different symptom from an empty list, and a different fix.
Getting traffic in from outside
ClusterIP is reachable only from inside the cluster. Three ways out, in increasing order of what you'd actually run in production:
NodePort opens the same port on every node, in the range 30000–32767 by default, and forwards it to the Service:
80:31584 means the Service's port 80 is also reachable at port 31584 on every node's own address. Useful for testing and bare-metal setups; awkward in production because clients need node addresses and a high port, and nodes come and go.
LoadBalancer asks the cloud provider for an external load balancer pointing at those NodePorts:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
api-lb LoadBalancer 10.96.184.22 203.0.113.77 80:31584/TCP 2m
Simple and expensive — it's one cloud load balancer per Service, each with its own bill and its own IP address. Ten services means ten of them.
Ingress is the answer to that expense. It's a single entry point that routes by hostname and path at Layer 7:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web
spec:
ingressClassName: nginx
rules:
- host: api.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api
port:
number: 80
One load balancer, many services, routing decisions based on the HTTP Host header and path — which is a reverse proxy, running inside the cluster, configured by Kubernetes objects instead of a config file. TLS termination usually lives here too.
The part that catches everyone once: an Ingress resource does nothing on its own. It's a description of desired routing, and something has to implement it — an ingress controller such as ingress-nginx or Traefik, installed separately. Apply an Ingress to a cluster with no controller and you get no error, no events, and no traffic. kubectl get ingress shows an empty ADDRESS column, which is the tell.
Triage: a Service that doesn't answer
Work outward from the pod, testing one layer at a time:
# 1. Does the pod itself serve? Bypasses Service, DNS, everything.
kubectl exec -it api-7d4b9c8f5-2xk4p -- curl -s localhost:8080/health
# 2. Does the pod's IP serve, from another pod? Tests the CNI.
kubectl exec -it redis-0 -- wget -qO- http://10.244.1.7:8080/health
# 3. Does the Service have endpoints? Tests selector and readiness.
kubectl get endpointslice -l kubernetes.io/service-name=api
# 4. Does the ClusterIP serve? Tests kube-proxy's rules.
kubectl exec -it redis-0 -- wget -qO- http://10.96.184.22:80/health
# 5. Does the name resolve to that ClusterIP? Tests cluster DNS.
kubectl exec -it redis-0 -- nslookup api
Each step eliminates one layer, and the first one that fails names the component to investigate. Step 1 failing is an application problem. Step 2 failing with step 1 passing is a CNI or network policy problem. Step 3 empty is labels or readiness. Step 4 failing with step 3 populated is kube-proxy. Step 5 is the one covered next.
Skipping to step 5 and concluding "DNS is broken" when the real problem is at step 3 is the most common wasted hour in Kubernetes debugging.
Practice
- Create a Deployment and a Service, then delete every pod. Watch
kubectl get endpointslicewhile they're recreated and note how long the endpoint list is empty. - Deliberately break the Service's selector by one character. Confirm the endpoint list empties, then predict what a client connecting to the ClusterIP will see — refused, or hung? Verify it.
- Set
targetPortto a port the container isn't listening on. Compare the symptom to exercise 2's, and write down the difference in one sentence each. - Open a long-lived connection from one pod to a Service, then scale the Deployment from 2 to 6. Confirm the existing connection stays on its original pod, and explain why.
- Apply an Ingress on a cluster with no ingress controller installed. Note exactly what feedback Kubernetes gives you (there isn't much), then install a controller and watch the
ADDRESScolumn change.
Exercise 4 is the one worth internalising. It's the reason a service can be scaled up during an incident with no effect on load distribution, and the reason gRPC and HTTP/2 clients in Kubernetes need either a proxy that understands request-level balancing or a headless Service and client-side balancing — a design decision that follows directly from what a ClusterIP mechanically is.
Sources
- Kubernetes, Service
- Kubernetes, Ingress
- Kubernetes, Cluster Networking
- Kubernetes, Virtual IPs and Service Proxies