A method for debugging network problems

Nobody reports a network problem accurately. What arrives is "the API is down," "the site is slow for some people," or a screenshot of an error dialog with no timestamp. The first job isn't fixing anything — it's converting that into a question with a testable answer.

Everything in this course so far has been a piece of the answer: addresses, routing, ports, DNS, TLS, proxies. This module is the order to use them in, and the discipline that keeps an investigation from turning into an hour of randomly restarting services.

Bisect the stack instead of walking it

The advice you'll usually hear is "work up the OSI layers from the bottom." It's not wrong, but it's slow, and it wastes time on layers that are almost never the problem — a server that has been serving traffic for six months does not have a cable fault.

Bisect instead. A request from a browser to an application passes through five stages, and each one can be tested independently, in about two seconds:

1. Name        →  does the name resolve, and to what?
2. Route       →  can a packet reach that address at all?
3. Port        →  is something listening there, and does it accept a connection?
4. TLS         →  does the encrypted session establish?
5. Application →  does the service answer correctly?

Test stage 3 first. If a TCP connection to the port opens, stages 1 through 3 are all fine and the problem is at 4 or 5 — you've eliminated more than half the stack with one command. If it doesn't open, drop to stage 1 and work forward.

nc -zv -w 3 api.example.com 443

One command, one bit of information, and the search space halves. That's the whole method.

The five commands, and what each one proves

Stage	Command	Proves, when it works
Name	`dig +short api.example.com`	The name resolves, and you can see to which address
Route	`ip route get <that address>`	The kernel has a path, and you can see the source address it will use
Port	`nc -zv -w 3 <host> 443`	The TCP handshake completes end to end — routing, firewalls, NAT and the listening socket all work
TLS	`openssl s_client -connect <host>:443 -servername <host> </dev/null`	Certificate and cipher negotiation succeed
Application	`curl -v https://api.example.com/health`	The service itself answers, with a status code you can read

Run them on the machine that's actually failing. A test from your laptop proves something about your laptop's path, which is frequently a different path with different firewall rules — and "it works from my machine" has never once resolved an incident.

Three questions to ask before touching a command

Who is failing, and who isn't? One user or everyone; one region or all of them; one pod or the whole deployment. A failure that affects a subset is a routing, DNS, or load-balancer problem far more often than a service problem, because a broken service usually breaks for everybody.

When did it start, and what changed then? Deployments, certificate renewals, DNS edits, firewall changes, and cloud security-group edits are the overwhelming majority of causes. If the answer is "about an hour ago," look at what happened about an hour ago before looking at packets.

Is it total or intermittent? This decides your tools. Total failure yields to nc and dig immediately. Intermittent failure needs sampling over time — mtr --report for path loss, ss -ti for per-connection retransmits, a tcpdump left running with a filter. A single successful curl proves nothing about a problem that happens one time in twenty.

"It works when I test it" is not evidence when the complaint contains the word "sometimes."

Test from both ends, and check the reverse path

A network path is two paths. Packets from client to server can take completely different links than the replies, and firewall rules are frequently asymmetric — a security group that allows inbound traffic but whose reply route was never configured produces the exact symptom of a connection that hangs after the SYN.

When you have access to both machines, capture from both. tcpdump on the server showing an incoming SYN with no matching reply localises the problem to the server's own stack or firewall. tcpdump on the server showing nothing at all localises it to the network in between, which is a different team and a different fix.

Write down what you've ruled out

An investigation that lasts more than a few minutes needs a note, because working memory is unreliable when three people are asking for updates:

14:02  dig +short api.example.com → 203.0.113.44   (name OK, correct address)
14:03  nc -zv -w3 203.0.113.44 443  → timed out    (port NOT reachable from app-02)
14:04  nc -zv -w3 203.0.113.44 443  from app-01 → succeeded  (works from a different host!)
14:06  ip route get 203.0.113.44 on app-02 → src 10.20.5.9 (different subnet than app-01)

Four lines, and the shape of the answer is already visible: it's not DNS, not the service, not the destination — it's something about traffic from app-02's subnet specifically. The next place to look is the firewall or security group rule that lists allowed source ranges.

That note is also what you hand over when the incident outlives your shift, and it's what stops a colleague from re-running the same three checks you already did.

Changing things is how a small outage becomes a large one

The strong pull during an incident is to do something — restart the service, flush the firewall, edit a DNS record, bounce the load balancer. Every one of those destroys evidence, and some of them create a second, unrelated failure on top of the first.

Before any change: capture the current state (ip route show, sudo nft list ruleset, dig's answer, the relevant config file) to a file. Change one thing at a time and re-test between changes — two simultaneous changes mean you won't know which one helped, and if the situation gets worse you can't tell which to revert. And never flush a firewall ruleset on a remote host to "test whether it's the firewall": that both opens the machine to the internet and can drop the SSH session enforcing it.

What this module covers

The next article is the single most useful piece of network debugging knowledge there is: the four ways a connection attempt can fail, and how each failure message points at a different part of the network. Getting that distinction right turns "the connection doesn't work" into "a stateful firewall between us is dropping packets silently" — before you've opened a capture.

After that comes triage for the two stages that break most often in practice, which are also the two nobody tests until they fail: name resolution and TLS.

Sources

Linux man-pages, connect(2) — the authoritative list of errors a connection attempt can return.
Linux man-pages, errno(3)