Skip to content

Flow control

A fast sender talking to a slow receiver is a specific, common failure mode: the network link has plenty of capacity, nothing along the path is congested, and yet the receiving application still can't keep up. Maybe it's a database driver on an underpowered instance, or a logging agent momentarily stalled by disk I/O. Left unmanaged, a sender that doesn't know any of this would simply keep pushing data until the receiver's buffer overflows and starts dropping segments — which, ironically, triggers exactly the retransmissions and slowdowns TCP exists to avoid. Flow control is the receiver's way of preventing that, by continuously telling the sender exactly how much room it actually has.

The receiver's buffer is the limit, not the network

Sliding window introduced the general mechanism: a moving range of the byte stream allowed to be in flight at once, capped by whichever of two limits is smaller. Flow control is specifically about the first of those two limits — the receive window (RWND) — and it exists because of a mismatch that has nothing to do with the network path at all. Incoming segments land in a fixed-size receive buffer before the application ever reads them out. If that buffer is 64 KB and the application is slow to call read(), the buffer fills regardless of how fast or idle the network link itself is. TCP's answer is for the receiver to advertise, in the window field of every segment it sends, exactly how much of that buffer is currently free — and for the sender to never have more unacknowledged data in flight toward that receiver than the most recently advertised number.

Why 65,535 bytes stopped being enough

The window field in the TCP header is 16 bits, which caps it at 65,535 bytes — workable in the era TCP was designed, when megabit links were fast, but a real bottleneck on any connection with both a generous amount of bandwidth and a non-trivial round-trip time. A rule of thumb makes the limit concrete: the most data usefully in flight on a connection is roughly bandwidth × round-trip time. A gigabit link with a 50ms round trip can theoretically carry about 6.25 MB of data in flight before the first ACK could possibly return — nearly a hundred times more than a 64 KB window allows. Capped at 65,535 bytes, that connection sits idle waiting for acknowledgments long before the link itself is anywhere near full, throttled by the window field's bit width rather than by the network or the receiver's actual buffer.

Window scaling, negotiated as a TCP option during the handshake, fixes this without changing the 16-bit field itself: both sides agree on a scale factor, and every window value sent afterward is interpreted as that raw field multiplied by 2 raised to the negotiated factor. Three-way handshake showed this option (wscale 7, wscale 8) appearing in the SYN and SYN-ACK — and critically, it can only be negotiated during the handshake. A packet capture that misses the handshake has no way to recover the real scale factor and can only show the raw, un-scaled 16-bit value from that point on, which is why a capture starting mid-connection sometimes shows window sizes that look implausibly small for an otherwise healthy, fast connection.

Watching the window change as a receiver empties its buffer

sudo tcpdump -i eth0 -n host 93.184.216.34 and port 5432 -c 5

09:15:01.100011 IP 10.0.5.12.51022 > 93.184.216.34.5432: Flags [P.], seq 1:1401, ack 1, win 501, length 1400
09:15:01.140550 IP 93.184.216.34.5432 > 10.0.5.12.51022: Flags [.], ack 1401, win 68, length 0
09:15:01.541002 IP 93.184.216.34.5432 > 10.0.5.12.51022: Flags [.], ack 1401, win 340, length 0
09:15:01.560884 IP 10.0.5.12.51022 > 93.184.216.34.5432: Flags [P.], seq 1401:2801, ack 1, win 501, length 1400
09:15:01.601115 IP 93.184.216.34.5432 > 10.0.5.12.51022: Flags [.], ack 1401, win 720, length 0

The server's window drops sharply right after the first segment arrives (win 68, down from a healthier baseline), then climbs back up over the next few hundred milliseconds with no new data from the client at all (win 340, then win 720) — this is the receiving application slowly draining its buffer by actually processing the data it already has, and the growing window is the receiver reporting newly freed space, entirely independent of anything the sender did. A window that shrinks and never recovers, by contrast, is the signature of a receiver that has stopped reading altogether.

The zero-window edge case, and how the sender finds out it's over

If a receiver's buffer fills completely, it advertises a window of zero — an explicit instruction to stop sending entirely. The sender obeys, which creates an obvious problem: the very next segment that would tell the sender the window has opened back up is itself blocked by that zero window, since flow control forbids sending unacknowledged data past the limit the receiver last advertised. If the receiver's own "window reopened" notification depended on the sender sending something first, both sides would deadlock.

TCP breaks this with window probing: after a zero window, the sender periodically sends a small probe segment — one byte, specifically permitted to bypass the zero-window restriction — purely to prompt a fresh ACK carrying the receiver's current window value. If the buffer's still full, the receiver just replies with another zero window and the sender waits and probes again; the moment the buffer has freed any space, the reply carries a non-zero window and normal sending resumes immediately.

Practical scenario: an ingestion pipeline that stalls in bursts

A metrics-ingestion service receives a steady stream of writes from application servers, but every few minutes throughput drops to near zero for several seconds before recovering on its own — visible to clients as intermittent request latency spikes, not outright failures. CPU and network utilization on the ingestion service both look unremarkable throughout.

A capture during one of the stalls shows the pattern:

sudo tcpdump -i eth0 -n host 10.0.9.44 and port 8125 -c 4

14:02:11.001100 IP 10.0.3.20.48812 > 10.0.9.44.8125: Flags [P.], seq 41201:42601, ack 900, win 2053, length 1400
14:02:11.041980 IP 10.0.9.44.8125 > 10.0.3.20.48812: Flags [.], ack 42601, win 0, length 0
14:02:14.042110 IP 10.0.3.20.48812 > 10.0.9.44.8125: Flags [.], seq 42601:42602, ack 900, win 2053, length 1
14:02:14.082450 IP 10.0.9.44.8125 > 10.0.3.20.48812: Flags [.], ack 42601, win 0, length 0

win 0 on the ingestion server's ACKs, followed a few seconds later by a one-byte window probe from the client (length 1), matches this article's description of a zero-window condition exactly. Since network and CPU utilization are both fine, the receive buffer filling isn't a capacity problem — it's an application-level one: something inside the ingestion service is falling behind on reading data out of its own socket buffer, most plausibly a downstream write (to disk, or to a database) that occasionally blocks the thread responsible for draining the socket. The network layer here is working exactly as designed; the useful next step is profiling the ingestion service's own read loop and its downstream write path, not touching anything at the network layer, since a wider receive buffer would only delay the same stall rather than fix its actual cause.

Practice exercises

  1. Explain, using the receiver's-buffer framing in this article, why a flow-control stall like the one above can happen even on a completely uncongested, low-latency local network — and why increasing the network's bandwidth wouldn't fix it.
  2. A packet capture shows a healthy-looking connection with a window value of 64 throughout, on a link with plenty of available bandwidth and a 40ms round-trip time. Using the scaling explanation above, list the two possible reasons this connection's real, effective window might be much larger than it appears — and one way to tell them apart if you had captured the full handshake.
  3. Walk through what happens, step by step, from the moment a receiver's buffer hits zero to the moment normal data transfer resumes, without looking back at the zero-window section.
  4. Explain why a sender is not allowed to simply ignore a zero-window advertisement and keep sending anyway "just in case the receiver actually has room" — what specifically would that risk, given what a receive buffer is for?

Sources