Sending Is Not Receiving in Reverse

Finding them requires giving up the assumption that sending is receiving played backwards. It is not, and the reason is structural: receiving is something that happens to your machine, and sending is something your machine chooses the timing of. Choice means policy, and all the policy in TCP lives on this side.

The obvious design

read receives. write sends. Symmetric names, symmetric behaviour: when the write returns, the data has gone.

Where it breaks

The write returns when your bytes have been copied into the socket’s send queue. That is all it means, and the emphasis belongs on all.

It does not mean nothing was sent. On an idle connection with room in both windows, TCP will usually push a segment out before the call returns, and your bytes really are on the wire. Nor does it mean something was sent: under any kind of pressure, the same call returns with every byte still sitting in memory. The point is not that the data has gone or has not gone — it is that the return of the call carries no information about it either way, and any reasoning you do based on when it returned is reasoning about the wrong event.

What happens to those bytes afterwards is TCP’s decision, made on its own schedule, according to rules with no counterpart on the receive side.

If the send queue is full, the behaviour depends on the socket: a blocking socket parks your thread until space appears; a non-blocking one — which is what your event loop is using — writes what it can and tells you how much, leaving you to keep the rest and try again when the descriptor reports writable. That, incidentally, is the other half of Post 08’s readiness interface, and the half most people forget exists until a slow client makes their memory usage climb.

What actually happens

Two limits, not one

Before sending anything, TCP works out how much it is allowed to have outstanding. There are two independent answers and it obeys the smaller.

The first is the receive window from Post 07: what the peer said it has room for. This protects the receiver from being overrun.

The second is the congestion window: TCP’s own estimate of how much the network between here and there can carry without collapsing. This protects the network, and nobody sent it to you — your kernel inferred it.

These are genuinely different things, and conflating them is the source of a lot of muddled reasoning. The receive window is a fact, transmitted. The congestion window is a guess, continuously revised.

How the guess is made

A new connection knows nothing about the path, so it starts deliberately small — about ten segments, roughly 14 kilobytes — and grows.

While things are going well it grows exponentially: each round trip, the window roughly doubles. This phase is misleadingly named slow start, and it is the opposite of slow; it is the fastest safe way to find the ceiling.

When loss appears — a gap in the acknowledgements — TCP concludes it has found the limit, cuts the window sharply, and switches to growing linearly, probing gently upward until the next loss. The sawtooth this produces is the characteristic shape of a bulk TCP transfer, and it is a control loop searching for a value it can never be told.

Now the consequence that matters for anyone serving web traffic.

A fresh connection can send about 14 kilobytes in its first round trip, 29 in the second, 58 in the third, 116 in the fourth. To deliver a 200-kilobyte response, the connection must complete four round trips before it has even been permitted to put the last byte on the wire.

And four is the optimistic figure, because the doubling assumes every segment is acknowledged individually. The receiver’s delayed acknowledgements from the next section mean it often is not, so the window grows more slowly than the arithmetic above suggests and real connections take longer.

On a path with a 50-millisecond round trip, four round trips is 200 milliseconds of pure protocol — with an idle network, a fast server, and a handler that returned instantly.

The forty milliseconds

Now the opening puzzle, which is a collision between two optimisations that are each correct alone.

On the sending side there is a rule against dribbling. If a sender has already sent a small amount of data that has not been acknowledged, and the application asks it to send another small amount, it holds the second one back and waits for the acknowledgement — then sends everything accumulated as one larger segment. The motivation was real and historical: a terminal session sending one keystroke per packet spends forty bytes of header to carry one byte of payload.

On the receiving side there is a rule against chattering. An acknowledgement carrying no data is pure overhead, so the receiver delays it briefly, hoping either to acknowledge two segments at once or to piggyback it on a reply its own application is about to send. On Linux that delay is adaptive, bounded at the low end by 40 milliseconds and at the high end by 200, and it moves within that range according to what the connection has been doing. Older stacks simply used a fixed 200 — which is why that number appears in so many bug reports written before about 2005, and why the symptom is sometimes remembered as a 200-millisecond stall rather than a 40-millisecond one.

Put them together with a handler that writes its response in two parts — say, headers and then body.

The first write goes out. The second is small, and there is unacknowledged small data outstanding, so the sender holds it. The receiver has the first piece, but has nothing to say and no data of its own to send, so it holds its acknowledgement. The sender is waiting for the acknowledgement. The receiver is waiting for something to acknowledge it with.

Neither is wrong. Nothing is broken. The connection sits there until the receiver’s delay timer expires, the acknowledgement is finally sent, and the sender immediately transmits the remaining bytes.

The fixes follow directly from the description. Disable the sender’s rule for this socket, which is what virtually every HTTP server does by default now. Or — better, because it is free — write the response in one call, with a vectored write or by assembling the buffer first, so there is never a small second write to hold back.

Down through the layers

Past TCP, the remaining steps are quick to state and each hides a queue.

IP looks up the route, fills in a header, and passes the packet down through the same firewall hooks the receive path used, in their outbound positions.

Then the queueing discipline: a queue between the network stack and the driver, and the place where traffic shaping, prioritisation, and fairness between flows actually happen. On a modern Linux system the default is specifically designed to keep this queue short, for reasons Post 11 is largely about.

Then the driver writes a descriptor into the transmit ring — the mirror of Post 03’s receive ring, with ownership flowing the other way — and rings a doorbell to tell the card there is work.

The card reads the buffer out of memory by DMA, frames it, and transmits it.

And then one more step that only exists because of ownership: the card raises a completion interrupt to say it is finished with the buffer. Until that moment the kernel must not free or reuse that memory, because the card still owns it and may not have read it yet. Post 03’s rule, holding in the opposite direction.

Segmentation, pushed downward

One complication worth knowing, because it will confuse a packet capture.

Cutting a stream into segments and writing a nearly identical header onto each one is exactly the sort of repetitive work a card can do. So the kernel often does not segment at all. It hands the driver a buffer of up to 64 kilobytes plus a note saying split this into pieces of this size and replicate the headers, and the card does the rest.

This is the transmit-side twin of the merging in Post 04, with the same motive: the per-packet cost is high, so handle fewer packets. And it has the same consequence — a capture taken on the sending host shows enormous packets that never existed on the wire, and checksums that look wrong because the card had not filled them in yet.

It also complicates the queueing discipline, which now sees one 64-kilobyte object where it used to see 45 packets it could interleave with someone else’s traffic. A great deal of subtle work has gone into limiting how much any single connection may have queued below TCP, precisely so that one bulk transfer cannot monopolise the transmit path and add latency to everything else.

See it for yourself

Everything above is in one command’s output, and this is the single most informative line in Linux networking.

ESTAB 0  1073728   10.0.0.7:8080   10.0.0.31:52418
   cubic wscale:7,7 rto:212 rtt:11.4/2.1 mss:1448
   cwnd:74 ssthresh:61 bytes_sent:48211904 retrans:0/19
   send 75.2Mbps pacing_rate 90.3Mbps delivery_rate 71.8Mbps

Read that as a story. The window is 74 segments, about 107 kilobytes, so this connection has grown well past its initial ten. The slow-start threshold below it means loss has been seen at least once and it is now in linear growth. There have been 19 retransmissions across 48 megabytes, which is unremarkable. And Send-Q at just over a megabyte means the application has written far more than TCP has been allowed to send — the application is not the bottleneck here; the path is.

Then look at the queue below the stack:

qdisc fq_codel 8003: root refcnt 2 limit 10240p flows 1024 quantum 1514
Sent 91847362 bytes 68214 pkt (dropped 12, overlimits 0 requeues 3)
backlog 0b 0p requeues 3
maxpacket 1514 drop_overlimit 0 new_flow_count 91 ecn_mark 0

And the cheapest experiment in this post: find a service that writes its response in two calls, join them into one, and measure again. If the number was forty milliseconds, it will not be.

What to carry forward

Receiving is reactive. A packet arrives, and the machine’s entire job is to keep up. There is no policy, only capacity, and the failure mode is dropping things.

Sending is a decision. TCP holds your bytes, estimates what the network will bear, waits for permission from the receiver, possibly waits for a timer, possibly waits for a card, and releases them on a schedule it computed. The failure mode is not dropping — it is delay, and delay in a place your profiler will never show you, because your write already returned.

Which sets up the last real post in the series. You now have every queue on the path, in both directions, and a way to observe each one. It turns out that is enough to explain essentially every networking failure you have ever seen — with one picture.