TCP Is a Lie Your Kernel Tells Beautifully

Where packets stop existing, why a read can return one and a half requests, and how the receive window pushes back.

Your parser has worked for a year. It reads from the socket, finds the end of the HTTP headers, reads the body, handles the request. Thousands of times a second, without incident.

Then you put the service behind a load balancer, or a client starts sending larger payloads, or you move it to a different network — and you start seeing truncated JSON. Not often. A fraction of a percent. And when you go looking, the client swears it sent the whole thing, the logs agree, and your code has not changed at all.

Nothing about your code changed. What changed is that the illusion TCP was selling you finally slipped, and you had been depending on it without knowing.

This post is about that illusion: how it is built, what it genuinely guarantees, and — the part that catches people — what it never promised and you assumed anyway.

The obvious design

TCP is described, everywhere, as a reliable ordered byte stream. Reasonable inference: what goes in comes out, in order, intact. The sender writes a message; the receiver reads a message.

Half of that is exactly right. TCP will deliver every byte, in order, or tell you the connection is broken. There is no scenario where you read byte 900 before byte 899, and no scenario where byte 899 is quietly missing.

The other half was never true.

Where it breaks

Here is the structural fact that explains the entire problem, and it is worth stating on its own line:

TCP sequence numbers count bytes, not packets.

Every byte of the connection has a position in one continuous numbering, and that numbering is the only thing the protocol tracks. A segment is not a unit of meaning — it is a chunk of that byte range that happened to travel together, sized by whatever was convenient at the moment it was sent.

So when your application calls write twice, the bytes are appended to a send queue, one after the other, and nothing anywhere records that there was a seam between them. Later, TCP decides how to cut that queue into segments, and it decides based on the maximum size the path supports, how much the network seems able to carry, whether small data is already unacknowledged, and whether the card is doing the segmenting. Two writes may leave in one segment. One write may leave in forty.

At the receiver the same thing happens in reverse. Segments arrive, their payloads are appended to the receive queue in sequence order, and your read takes however much happens to be sitting there at the moment you ask.

Three requests, written back to back onto one connection.

Fig. 1One stream, cut three times by three parties who never consulted each other. The application chose the top boundaries, TCP chose the middle ones according to what the path supports, and your reads landed on the bottom ones according to what happened to be queued when you asked. The dashed lines are the only boundaries that carry meaning, and nothing below them respects those.

And that is why it worked for a year

On a local network, with small messages, one write usually does become one segment, and one segment usually does arrive alone, and one read usually does return exactly it. The illusion holds — not because it is guaranteed, but because you never gave it a reason to break.

Then something changes the segmentation. A payload grows past the maximum segment size and is split. A proxy in the path reassembles and re-splits the stream on different boundaries. A client sends two requests back to back and they coalesce. Traffic gets busy enough that segments start being merged on receipt, per Post 04.

The bug was always there. It was waiting for the segmentation to change.

What actually happens

Reassembly

Each arriving segment carries the sequence number of its first byte. The socket knows the number it is expecting next.

If they match, the payload is appended to the receive queue and the expected number advances by the payload length. Then the kernel checks the out-of-order queue: does the new expected number match anything parked there? Often it does — a gap has just been filled — and several segments’ worth of data moves across at once.

If the arriving segment is ahead of what is expected, a segment was lost or reordered. The payload is parked in the out-of-order queue and the receive queue does not advance, because handing your application byte 3000 before byte 2000 would break the one promise that matters.

If it is behind, it is data already received — a retransmission of something whose acknowledgement went missing — and it is discarded, though the acknowledgement is sent again so the sender stops.

Acknowledgement and repair

Acknowledgements are cumulative: an acknowledgement of byte 5000 means I have everything below 5000, not I have the segment starting at 5000. One acknowledgement can therefore confirm a great deal at once, and a lost acknowledgement is harmless if a later one arrives.

The sender keeps everything unacknowledged in its send queue, because it may need to send it again. It repairs loss two ways. It runs a timer, and if an acknowledgement does not arrive in time, it retransmits — the slow path, because the timer must be conservative. Or it notices the receiver repeatedly acknowledging the same byte number, which means segments are arriving but the gap is not closing, and retransmits immediately without waiting for the timer.

An extension lets the receiver say not just I have everything below 5000 but also and separately, I have 6000 through 7448, so the sender can resend only the hole rather than everything after it.

None of this is visible to your application. A retransmitted segment appears in your receive queue as ordinary bytes in the right place. The only trace is time.

The window: a queue’s fill level, sent over the network

Now the part that deserves more attention than it usually gets.

Every segment the receiver sends carries a window: the amount of free space currently in its receive queue. It is not advice. The sender is not permitted to have more than that many unacknowledged bytes in flight.

Follow the consequences.

Your application stops reading — it is blocked on a database, or doing something expensive, or simply descheduled. Bytes keep arriving and keep being appended to the receive queue. The queue fills. The free space falls. Every acknowledgement your kernel sends carries a smaller number, and the sender slows down to match.

Eventually the free space reaches zero. Your kernel sends a window of zero, which means stop. And a machine that may be on another continent stops transmitting — not because of anything the network did, but because of the fill level of a queue in your RAM.

The sender then sits in a loop poking the connection occasionally to ask whether the window has opened, because if the message announcing the reopening were lost, both sides would wait forever.

And the chain continues past the sender. With its window closed, the sender’s own send queue stops draining, fills up, and eventually its write call blocks. An application somewhere else is now paused because your application stopped reading. That is a single chain of backpressure running from your event loop, through your kernel, across the network, through another kernel, into someone else’s code.

Framing is your problem now

TCP destroyed your message boundaries. Every protocol built on it has to invent them again, and there are exactly three ways.

Say how long it is. Send a length, then that many bytes. Nearly every binary protocol does this. It is unambiguous and easy to parse.

Use a delimiter. Agree on a byte sequence that cannot appear in the content, and scan for it. This is how HTTP knows where the headers end: a blank line.

Close the connection. The message ends when the stream does. This works exactly once per connection, which is why it went out of fashion.

HTTP/1.1 uses the first two, in different places, which is worth noticing because it is unusual: a delimiter for the request line and headers, then a declared length for the body — either a Content-Length or the chunked encoding, which is a length prefix per chunk.

So a correct reader over TCP has exactly one shape. Read whatever is available. Append it to a buffer. Ask whether the buffer now contains a complete message. If yes, take it out and repeat the question, because there may be another. If no, read more. A parser that assumes one read is one message is not a parser — it is a coincidence that has not failed yet.

Beginning and ending

Two smaller pieces, for completeness.

A connection opens with three messages because both directions need to agree on where their byte numbering starts, and each side’s starting number must be acknowledged by the other. Those numbers are randomised, which is a security property: if they were predictable, anyone could inject bytes into someone else’s connection.

Closing is separately directional. Each side sends a marker saying I have no more data, which the other acknowledges. Between the two, the connection is half-open and perfectly usable in the remaining direction — which is what lets a client signal end-of-request and then wait for a response.

Afterwards, the side that closed first holds the tuple in a waiting state for a minute or so. Two reasons: a straggling duplicate from the old connection must not be mistaken for data on a new connection reusing the same tuple, and the final acknowledgement must be retransmittable if it went missing. It is not a leak and it is not tunable away safely. It is also why, given the choice, you would rather the client closed first than the server — a server holding tens of thousands of waiting tuples is holding tens of thousands of entries in the table from Post 06.

See it for yourself

You can watch the window close.

Start a server, connect a client that sends continuously, and make the server stop reading — pause the process with a signal, or put a long sleep in the handler. Then watch the connection.

Run thisLinux

ss -tin state established

Recv-Q Send-Q  Local Address:Port   Peer Address:Port
93184 0           10.0.0.7:8080       10.0.0.31:52418
 cubic wscale:7,7 rto:204 rtt:3.1/1.5 mss:1448
 rcv_space:65495 bytes_received:9441280 segs_in:6521

Recv-Q climbing while the reader is paused. When it stops climbing, the advertised window has reached zero and the sender has stopped. Resume the reader and the whole thing drains in one burst.

Do not expect that number to stop at a round figure. The limit is on the memory the socket may use, not on the payload it may hold, and each queued segment carries kernel bookkeeping alongside its bytes. So a socket allowed 128 kilobytes stalls somewhere short of that, at a point that depends on how large the arriving segments were. The ceiling it is approaching comes from here:

Run thisLinux

cat /proc/sys/net/ipv4/tcp_rmem

4096	131072	6291456

Three numbers: the minimum, the default a new socket starts with, and the maximum the kernel will grow one to. The middle number is the ceiling in the first output, and the kernel will raise it toward the third if the connection proves it can use the space.

And for the framing lesson, the cheapest demonstration is the one that costs you nothing: take whatever parses your protocol and feed it the same input one byte at a time. If it still works, it is a parser. If it does not, you have found the bug before production did.

What to carry forward

Your bytes are in a queue, in order, guaranteed complete. The packets that carried them no longer exist in any form, and the message boundaries went with them.

But your application still has not run. Something has to wake it, and on a server with fifty thousand connections, how it gets woken is the difference between a design that scales and one that does not.

That mechanism — and the last link in the chain that began with a hardware interrupt — is next.

What this post simplified

The receive window is not the only limit on how fast a sender may send, and on a healthy network it is rarely the binding one. The other limit is the sender's own estimate of what the network can carry, and it is the subject of Post 10.
I described the window as free space in the receive queue. The kernel also grows and shrinks the queue itself based on measured throughput, so the advertised window moves for reasons that have nothing to do with your application's reading speed.
Delayed acknowledgement means an acknowledgement is often not sent immediately, which changes the timing of everything described here and interacts badly with one specific sender-side algorithm. Post 10 has the collision.