The Whole Journey, Told Once, Slightly Wrong

This series is about the second box.

Not all of it. Nothing here is about DNS, or routing protocols, or what happens to a packet while it is in flight between two machines. Those are other people’s books. This is about what happens on one machine, in the tens of microseconds between a signal arriving at a network card and your function running — and then the same distance back out again.

That story has about a dozen stages. This post tells all of them, in order, once, quickly.

It is also wrong. Not vague, not merely incomplete: in at least seven specific places it says something that is flatly false. That is deliberate. At the end I will list the falsehoods, and each one turns out to be a later post in the series. The rest of this series is nothing but going back and fixing the lies told here.

So read this one for the shape. Do not try to hold the details — they all come back.

Arrival: from the wire to a socket

A signal becomes a frame

At the edge of the machine there is a physical connection — a pair of copper wires, a fibre, an antenna — and on it, a continuously varying electrical or optical signal. Not bits. A voltage.

The first chip in the path, the PHY, turns that signal into bits. It recovers the clock from the signal itself, samples at the right moments, and undoes the line code — the agreed scheme for representing bits as voltage transitions, chosen so that long runs of identical bits still produce enough edges to keep both ends synchronised.

The second block, the MAC, turns those bits into a frame. It watches for the preamble that marks a frame’s start, reads off the bytes until the frame ends, and then checks the trailing checksum. If the checksum fails, the frame is dropped right there and a counter increments; nothing above ever learns it existed. If it passes, the MAC checks the destination address. Is this frame addressed to this card, or to everyone, or to a multicast group this card was told to care about? If none of those, it is dropped too.

What emerges is a buffer of bytes that the card is confident is a real, undamaged, correctly-addressed Ethernet frame.

The CPU has not been involved at all.

The frame is written into memory, by the card

Here is the first place where the obvious mental model is wrong, and it is worth slowing down for, because the correction runs through the whole series.

The card does not hand the frame to anyone. It does not call the kernel. It does not wait to be asked.

Long before this frame arrived, the driver allocated a set of empty buffers in main memory and wrote their addresses into a structure the card can read: a ring of descriptors, each one saying here is an empty buffer, this many bytes long, you may use it. The card keeps an index into that ring. When a frame arrives, the card takes the next descriptor, writes the frame directly into that buffer over the bus, marks the descriptor as used, and advances its index.

Main memory is written by a device, without the CPU executing a single instruction. This is direct memory access, and the reason it exists is arithmetic: a 10-gigabit link carrying full-size frames delivers about 812,000 frames every second. Copying each one with the CPU would consume the machine.

Notice also what the ring really is. It is a queue with a fixed number of slots, shared between something that fills it (the card) and something that drains it (the kernel). If the filling outpaces the draining, the card runs out of descriptors and drops frames — silently, in hardware, before software has any idea. This is the first queue in the story and it will not be the last.

The card raises an interrupt

Now, finally, the CPU is told.

The card asserts an interrupt. Whatever the processor was doing — running your handler, running someone else’s process, idling — it stops, saves just enough state to come back, and jumps to a function the driver registered at boot.

That function does almost nothing. It notes that there is work waiting, asks the kernel to schedule the real processing for slightly later, and returns. The whole thing is over in well under a microsecond.

The restraint is the point. While an interrupt handler is running, the machine is in a strange and narrow state: it cannot sleep, it cannot take its time, and other interrupts may be held off behind it. So the handler’s only job is to hand the work to something that runs under normal rules.

The kernel drains the ring

A moment later — a different moment, in a different execution context — the kernel comes back and does the actual work.

It walks the ring, and for each frame the card deposited it allocates a small struct that becomes the packet’s identity for the rest of its life inside the kernel. That struct does not contain the bytes. It points at the bytes, carrying a pointer to the start of the buffer and another to the start of whatever layer is currently being examined.

This matters more than it sounds. As the packet moves up through the layers, nothing is copied and nothing is trimmed. A pointer moves forward past the Ethernet header, then past the IP header, then past the TCP header. “Stripping a header” is a pointer add.

Every layer asks the same question

What follows is the same move, performed once per layer.

Look at the header in front of you. Read the one field that says what is inside. Use it to decide who handles this next. Move the pointer past your header. Hand it on.

The Ethernet header has a field saying what protocol it carries — IPv4, IPv6, ARP. It says IPv4, so the packet goes to the IP code. The IP header is checked for a sane version and length, its checksum verified, and its destination address examined: is this address one of mine? It is, so the packet is for local delivery rather than forwarding. The IP header has a field saying what protocol it carries. It says TCP, so the packet goes to the TCP code.

Along the way the packet passes several points where the firewall gets to inspect it, accept it, rewrite it, or drop it — but those are hooks bolted onto this same path, not a different path.

The kernel finds your socket

TCP’s first question is: which connection is this?

It takes four values — the source address, the source port, the destination address, the destination port — hashes them together, and looks the result up in a table. That table maps connections to sockets. If there is a match, this packet belongs to an established connection and the kernel now has the exact socket struct that your accepted connection refers to.

If there is no match, the kernel tries a second table: the listening sockets. A match there means this is the first packet of a new connection, and the handshake begins. A match in neither means nobody is home, and the kernel replies with a reset.

That socket struct is the thing your file descriptor actually points to. It is not a magical channel. It is a struct, and its most important contents are two queues: one holding bytes that have arrived and not yet been read, one holding bytes you have written that the other end has not yet acknowledged.

A stream is reassembled

TCP’s second question is: where does this go?

Every TCP segment carries a sequence number saying where its payload sits in the overall stream. If this segment is the next one expected, its payload is appended to the socket’s receive queue and the connection moves on. If it arrived early — because an earlier segment was lost or took a different route — it is parked in a separate holding area until the gap is filled. If it duplicates something already received, it is discarded.

And then TCP does one more thing, which is the part almost nobody notices. It looks at how much free space is left in that receive queue, and it puts that number in the next packet it sends back. That number is the sender’s instruction about how much more it is allowed to send.

This is backpressure, and it is a message sent over the network, derived from the fill level of a queue in memory on your machine. If your application stops reading, the queue fills, the advertised number falls to zero, and a machine possibly thousands of kilometres away stops transmitting.

Delivery: from the socket to your function

Your process is woken

The receive queue has just gone from empty to not-empty. That transition is an event, and the socket keeps a list of who wants to know about it.

Maybe a thread of yours is blocked in a read call, parked and unrunnable. Then the kernel marks it runnable, and the scheduler will get to it shortly.

More likely, for a server, that list contains an entry belonging to an epoll instance — the kernel object your runtime’s event loop is built on. The kernel runs a small callback which moves this socket onto that epoll instance’s ready list. And when your event loop next calls into the kernel to ask which of my several thousand connections have something for me, the answer is already computed, sitting in a list, waiting.

The chain from a hardware interrupt to a callback in your runtime is now complete. It never went through your code once.

Your handler runs

Your event loop wakes, sees a descriptor is readable, and reads. That call copies bytes out of the kernel’s receive queue into a buffer your process owns, and returns a count.

Now — and only now — does anything resembling HTTP exist.

Your HTTP parser looks at those bytes and tries to find a request line, then headers, then the blank line that ends them, then a body whose length the headers described. If the bytes are not all there yet, it keeps what it has and waits for the next read. If two requests arrived together it must find the boundary between them itself.

Once a complete request is assembled, your framework matches the path against its route table, builds whatever request object it likes to build, and calls your function.

The request has arrived.

The return trip, which is not a return

You build a response and call something that ends in a write to the socket.

That call copies your bytes into the socket’s send queue and returns. Whether anything was transmitted on the way is not something it tells you — often TCP does push a segment out before the call returns, and just as often it does not. Either way, by the time control is back in your handler, the bytes are the kernel’s problem and not yours, and they may not leave for some time.

What happens afterwards is TCP’s decision, not yours. It works out how much it is permitted to send — limited by what the receiver said it could accept, and separately by how much the network has recently seemed able to carry — chops the stream into segments, attaches headers, and hands each one down. IP adds its header and finds the route. The packet lands in a queueing discipline, one last queue where traffic shaping and prioritisation happen. The driver writes a descriptor into the transmit ring and rings a doorbell. The card reads the buffer out of memory by DMA, the MAC frames it, the PHY puts it on the wire, and eventually an interrupt tells the kernel the buffer can be reused.

Same components, opposite direction. But it is emphatically not the same journey backwards, for the simple reason that arrival is something that happens to your machine and departure is something your machine chooses the timing of.

See it for yourself

Everything above is abstract right up until you look at a real socket. Two of the queues in this post are printed by default in a command you may already have typed a thousand times without reading those columns.

State   Recv-Q  Send-Q     Local Address:Port      Peer Address:Port
LISTEN  0       4096             0.0.0.0:8080            0.0.0.0:*
ESTAB   0       0               10.0.0.7:8080          10.0.0.31:52418
ESTAB   1448    0               10.0.0.7:8080          10.0.0.31:52420

On a listening socket those columns mean something different and more interesting: Send-Q is the configured size of the queue of finished handshakes waiting to be accepted, and Recv-Q is how many are sitting in it right now. A non-zero Recv-Q on a listening socket means connections are completing faster than your application is accepting them. That is a queue filling up, and you can watch it happen.

Seven lies

Here is what I just told you that is not true.

One. “The card hands the frame to the kernel.” It does no such thing. It writes into memory the kernel handed it in advance, and every interesting property of that arrangement — how many buffers there are, who owns one at any instant, what happens when they run out — is invisible in the sentence I wrote. Fixed in Post 03.

Two. “The card raises an interrupt and the kernel processes the packet.” True for the first packet of a burst. False for the next several thousand. A server under real load takes dramatically fewer than one interrupt per packet, and the mechanism that makes that possible is the most elegant thing in the entire stack. Fixed in Post 05.

Three. “The kernel does this.” Which kernel? I used the word to cover at least four different execution contexts, with different rules about what they may do, whether they can be interrupted, and whether they can wait. Treating them as one actor makes the rest of the stack impossible to reason about. Fixed in Post 02.

Four. “The port identifies the socket.” I was careful to say four values, but it is worth stating plainly, because the folk version is so widespread: a port number does not identify a connection. Ten thousand clients connected to port 443 produce ten thousand distinct sockets, and what tells them apart is the whole four-tuple. Fixed in Post 06.

Five. “TCP puts the bytes in order and your application reads them.” It also decides whether to accept them at all, when to acknowledge them, and how much the sender may send next — and it communicates that last decision using the free space in a queue as the message. Fixed in Post 07.

Six. “Your application decodes the packet.” By the time your code is involved there is no packet. There is a stream of bytes with no markings on it. One read may hand you half a request, or two and a half requests. Every message boundary that TCP dissolved, your parser has to rebuild. Fixed in Post 07 and Post 09.

Seven. “The response goes back the same way in reverse.” Your write returned before anything was sent. What happens after it returns is where essentially all response latency lives, and it is governed by rules that have no counterpart on the receive side. Fixed in Post 10.

What to take from this

Not the details. The shape.

If you remember one structural fact from this post, make it the one the diagram was drawing: this is not a pipeline of functions calling each other. It is a ladder of queues, and at every rung something fills the queue and something else — running at a different time, under different rules, sometimes on a different core — drains it.

That is why the stack behaves the way it does under load. That is why latency appears in places you did not write any code. And it is why, when something goes wrong, the useful question is almost never “which function is slow” but “which queue is full, and which side of it is the problem.”

The next post asks the question that makes the rest of the ladder legible: when your code is not running, who is?