Three-way handshake (SYN → SYN-ACK → ACK)
TCP vs UDP comparison captured a handshake in passing — three lines of tcpdump output with no payload, [S], [S.], [.], before any real data moved. This article takes those three lines apart. The handshake looks like a formality, three messages just to say hello, but every one of them is doing real work: agreeing on starting sequence numbers, confirming both hosts are actually listening, and establishing the connection state that every later mechanism in this module — the sliding window, flow control, congestion control — depends on already existing.
Why a handshake exists at all
Recall from the comparison article that UDP needs none of this: a UDP socket sends a datagram to a destination it's never contacted, and that's the whole interaction. TCP can't get away with that, because everything downstream of the handshake assumes both sides know two things about each other that neither side can simply guess: what sequence number the other side's byte stream is going to start counting from, and whether anyone is actually listening on that destination port at all. Skip the handshake and a sender has no way to know if its data even reached an active application, let alone whether the receiver can make sense of the sequence numbers attached to it.
The three-way exchange is the minimum number of messages that lets both sides establish and confirm this shared state in one round trip: one side proposes a starting sequence number, the other proposes its own and confirms the first, and the first side confirms the second. Two messages wouldn't be enough — the second side would have no way of knowing its own proposal was received — and a fourth message combining the last ACK with data is exactly what TCP allows in practice, since there's nothing in the specification stopping the final ACK from carrying application data already.
The exchange, message by message
Client (10.0.5.12) Server (93.184.216.34)
| |
| SYN seq=x |
| -----------------------------------------> |
| |
| SYN-ACK seq=y, ack=x+1 |
| <----------------------------------------- |
| |
| ACK ack=y+1 |
| -----------------------------------------> |
| |
[ESTABLISHED] [ESTABLISHED]
- SYN. The client picks an Initial Sequence Number (ISN) — call it
x— and sends a segment with theSYNflag set andseq=x. This says: "I want to open a connection, and I'm going to count the bytes I send starting fromx." No application data rides on this segment; it exists purely to propose a sequence space. - SYN-ACK. If a process is listening on the destination port, the server responds with both
SYNandACKset:seq=y(its own, independently chosen ISN) andack=x+1. Theack=x+1is the acknowledgment of the client's SYN — TCP treats a SYN as consuming one sequence number even though it carries no data, which is why the acknowledged value isx+1rather thanx. The server's ownSYNproposes its ISN,y, the same way the client's did. - ACK. The client acknowledges the server's SYN with
ack=y+1, and the connection moves toESTABLISHEDon both ends. From here on, every segment carries a sequence number counting up from the ISN each side chose, and every acknowledgment counts up from the other side's ISN.
Both sides pick their own ISN independently — TCP connections are full-duplex from the start, and the client's byte stream to the server and the server's byte stream back to the client are numbered completely separately. This is worth stating plainly because it trips people up: there's no single "the sequence number" for a connection, there are two, one per direction, and each one only ever increments relative to its own starting point.
Why the ISN isn't just zero
An early, much simpler design might start every connection's sequence numbering at 0. Real TCP implementations don't, and the reason is security, not correctness. If ISNs were predictable — sequential, or derived from something an attacker could observe or guess — an off-path attacker who knows or guesses a connection's four-tuple (source IP, source port, destination IP, destination port) could inject forged segments into an existing connection, or complete a spoofed handshake without ever seeing the server's SYN-ACK, by guessing the sequence number the real client would have received. RFC 9293 requires ISNs to be generated so they're effectively unpredictable to an outside observer, and modern stacks derive them from a combination of a timer and a per-connection secret hash, specifically to close off that class of attack. This is also part of why TCP is comparatively hard to spoof, a point TCP vs UDP comparison already flagged in the abstract: guessing a valid in-window sequence number for an established connection is a real obstacle, not a formality.
Capturing a handshake and reading each field
A capture makes the abstract exchange concrete. On a Linux host:
sudo tcpdump -i eth0 -n host 93.184.216.34 and port 443 and 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0' -c 3
10:14:02.001100 IP 10.0.5.12.53211 > 93.184.216.34.443: Flags [S], seq 2749103881, win 64240, options [mss 1460,sackOK,TS val 991823001 ecr 0,nop,wscale 7], length 0
10:14:02.038420 IP 93.184.216.34.443 > 10.0.5.12.53211: Flags [S.], seq 590182734, ack 2749103882, win 65160, options [mss 1440,sackOK,TS val 3820019271 ecr 991823001,nop,wscale 8], length 0
10:14:02.038650 IP 10.0.5.12.53211 > 93.184.216.34.443: Flags [.], ack 590182735, win 64256, options [nop,nop,TS val 991823009 ecr 3820019271], length 0
The tcp[tcpflags] & (tcp-syn|tcp-ack) != 0 expression filters to only segments carrying the SYN or ACK flag — a narrower, more deliberate capture than grabbing every packet on the port, and useful whenever the handshake itself, not the data that follows, is what's under investigation. Reading the three lines against the diagram above:
- Line 1:
Flags [S],seq 2749103881— the client's SYN, proposing ISN2749103881. Theoptionsfield is where the handshake does more than the seq/ack numbers alone show:mss 1460is the client announcing the largest segment it's willing to receive (covered in full in MSS),wscale 7is negotiating a window-scaling factor so the 16-bit window field can represent more than 65,535 bytes (see Sliding window), andsackOKadvertises support for selective acknowledgment. - Line 2:
Flags [S.]— the combined SYN-ACK —seq 590182734is the server's own ISN, andack 2749103882confirms the client's SYN (2749103881 + 1). The server replies with its own MSS and window-scale options, which can differ from the client's; each side's option applies only to what it can accept. - Line 3:
Flags [.],ack 590182735— the client's final ACK, confirming the server's SYN (590182734 + 1), withlength 0— this segment carries no data, though in practice a client is free to attach its first request to this same segment instead of sending a bare ACK, and modern stacks with TCP Fast Open do exactly that on repeat connections to the same host.
Every value above — sequence numbers, window sizes, timestamps — will be different on your own capture; ISNs in particular are randomized per connection by design, as the previous section explained.
What happens when nobody's listening
The handshake also has to handle the case where the destination port has no listening process at all, and it's worth seeing what that looks like, since it's easy to confuse with a firewall drop:
10:20:11.100221 IP 10.0.5.12.54011 > 10.0.5.20.9999: Flags [S], seq 118820033, win 64240
10:20:11.100889 IP 10.0.5.20.9999 > 10.0.5.12.54011: Flags [R.], seq 0, ack 118820034, win 0
An immediate RST, ACK — a reset — is the host actively saying "nothing is listening on this port," and it arrives fast, typically within a few milliseconds on a LAN, because the target's kernel responds the moment it finds no socket bound to that port. Contrast that with a SYN sent to a port a firewall silently drops: no reset ever arrives, and the client's connection attempt sits waiting until its own retransmission timer gives up, which takes much longer — often tens of seconds by default. That timing difference — a fast, explicit refusal versus a long, silent wait — is frequently the single fastest signal for telling "closed port" apart from "filtered port" during a live investigation, without needing a port scanner at all.
Practical scenario: a backlog filling up under a SYN flood
A public-facing API starts refusing new connections during a traffic spike, while existing connections keep working normally. curl from an unaffected network times out during connection setup specifically — not during the HTTP request itself:
A read-only check of the server's kernel-level SYN backlog is the first place to look, since a stalled connection attempt — as opposed to a stalled request on an already-open connection — points squarely at the handshake stage:
A large, saturated count of connections stuck in SYN-RECV — the server has sent its SYN-ACK and is waiting for the final ACK that never comes — is the signature of a SYN flood: an attacker (or, less maliciously, a badly behaved client population) sends a burst of SYNs, often with spoofed source addresses, and never completes the handshake. Each half-open connection consumes a slot in the fixed-size backlog queue until it times out, and once that queue is full, the kernel starts silently dropping new SYNs — legitimate clients experience exactly the connection-timeout symptom above, indistinguishable from the outside from a firewall dropping their traffic.
Mitigating a SYN flood touches a production kernel setting — treat it like one
The two most common mitigations, increasing net.ipv4.tcp_max_syn_backlog and enabling net.ipv4.tcp_syncookies, are effective and widely deployed, but they're still kernel-level networking changes. Confirm the current values first (sysctl net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_syncookies), change them on a test host or during a maintenance window if possible, and re-check the SYN-RECV count afterward rather than assuming the change worked. SYN cookies in particular trade away some of the connection options negotiated in the handshake (window scaling can be affected under cookie fallback), which is a real behavior change, not a free fix.
SYN cookies solve the backlog-exhaustion problem cleverly: instead of allocating backlog state for every SYN it sees, the server encodes what it needs to remember directly into the sequence number of its own SYN-ACK, and only allocates real connection state once the final ACK arrives proving the handshake is genuine — a half-open, never-completing connection under a cookie-based defense costs the server almost nothing to have offered.
Practice exercises
- Run the filtered capture from this article (
sudo tcpdump -i <interface> -n host <ip> and port 443 and 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0') against a real HTTPS site and identify each side's ISN and MSS option from the output. - Explain, using this article's description of how a reset is generated, why a fast
RSTand a long silent timeout are different failure signatures for the same symptom ("I can't connect") — and which one tells you the packet definitely reached the target host. - A teammate proposes an application-level "connection established" acknowledgment sent as a normal data message right after the TCP handshake completes, arguing it makes the client's connection more reliable. Explain what the TCP handshake already guarantees that this extra step would duplicate, and describe one thing it would not duplicate — something only the application layer can actually confirm.
- Using the
sysctlvalues referenced in the SYN-flood scenario, explain why raisingnet.ipv4.tcp_max_syn_backlogalone, without enabling SYN cookies, still leaves a server vulnerable to a large enough flood — what does the backlog size actually limit, and what does it not solve?