Where the Model Bends

One: inside a container

Start here because it is where most code now runs, and because the most common mistake it causes is a diagnostic one.

What changes. Very little, and specifically not the parts you might expect. A container is not a virtual machine. It shares the host’s kernel, so the driver, the ring, the interrupt handling, the softirq processing and the entire TCP implementation are the same code, in the same contexts, with the same counters. Nothing below the socket is duplicated.

What is duplicated is the configuration: a network namespace gives the container its own interfaces, its own routing table, its own firewall rules and its own socket tables. And to connect that namespace to the outside, the usual arrangement is a virtual cable — a pair of interfaces where anything sent into one comes out of the other, with one end inside the container and the other attached to a bridge on the host.

What the model predicts. A virtual cable is a boundary between two things, and this series has one prediction about boundaries: there is a queue on it.

There is. Each end of that pair has its own queueing discipline, its own backlog, and its own ability to drop packets. Which means your ladder from Post 11 just got longer, and every counter on it now exists at least twice: once on the host’s physical interface, once on the container side of the pair.

That is the diagnostic mistake. The drops are on the interface you are not looking at. Running ip -s link on the host shows a healthy physical NIC while packets are being discarded a few microseconds later on a virtual device that nothing is monitoring.

The second prediction comes from Post 06. If addresses are being translated — and in most container networking they are — then something is rewriting the tuple before the socket lookup happens, which means something is keeping a table of translations. A table is finite.

262144

Compare that against the current count in /proc/sys/net/netfilter/nf_conntrack_count. A service making very many short-lived outbound connections can fill it, and the failure looks like a network problem rather than a table overflow, which is why it costs people so many hours.

What stays true: everything from Post 03 through Post 08, unchanged.

Two: with encryption

What changes. A layer appears between the socket and your parser.

What the model predicts. Post 07’s lesson was that a layer which chops data into units destroys the boundaries of the layer above it, and the layer above has to rebuild them. Encryption adds a layer that does exactly that, so we should expect a third round of framing.

Correct. TLS does not encrypt a stream; it encrypts records of up to sixteen kilobytes. A record can only be decrypted once it has arrived completely, because its integrity check covers the whole of it.

That produces a consequence worth remembering, because it is a reliable source of bugs. Your event loop is told the socket is readable. Bytes really have arrived. You attempt to read the decrypted stream and get nothing, along with a request to try again later — because what arrived was part of a record, and part of a record is not decryptable into anything.

Readable is not the same as decryptable. Any code that assumes a readiness notification implies available application data will eventually stall, and it will stall under load, when segmentation changes — which is the same trap as Post 07, one layer up.

The other predictions follow the same way. Records mean a third flavour of head-of-line blocking. Decryption means a copy, which is why the record layer has gradually been pushed down into the kernel and, on some hardware, into the card — an ownership optimisation of precisely the sort Post 03 described. And the handshake means round trips, which is the time_appconnect figure in Post 09’s curl output and part of why Post 11’s cold-connection penalty is as large as it is.

Three: over QUIC

The radical one, because it moves a boundary this series treated as fixed.

What changes. HTTP/3 does not run on TCP. It runs on QUIC, which runs on UDP, and UDP does almost nothing: no connection, no ordering, no reassembly, no retransmission, no congestion control. All of that still has to happen, so all of it moves into a library in your process.

What the model predicts. The queues do not disappear. They move.

They do. The socket receive queue is now a UDP socket’s queue holding whole datagrams. Reassembly buffers, the out-of-order queue, retransmission state, and the congestion window are all in your process’s heap, managed by the QUIC library, subject to your garbage collector and your scheduler rather than the kernel’s.

Post 06’s five-tuple stops being the connection’s identity, replaced by a connection ID carried in the packet header. It has to be in the clear — it is what selects the keys, so it cannot itself be encrypted — and that is the one field a receiver can read before it has decided anything else. Which means the thing Post 06 presented as structural turns out to have been a design choice, and changing it buys something real: a connection survives its addresses changing, so moving from Wi-Fi to cellular no longer kills every connection on your phone.

And Post 11’s unfixable problem gets fixed. QUIC carries independent streams, so a lost packet stalls only the stream whose data it carried. The transport head-of-line blocking that HTTP/2 could not solve — because it was below HTTP/2 — is solved by removing the assumption that a connection is one ordered stream.

What it costs, and the model predicts this too. Post 05’s whole argument was that per-packet cost dominates at high rates, and QUIC raises per-packet cost substantially: every datagram crosses into userspace, and cryptographic protection now extends over most of the transport header rather than stopping at the payload. So the same amortisation techniques have had to be reinvented on top of UDP — batching many datagrams per system call, and segmentation and merging offloads for UDP that mirror the TCP ones from Post 04 and Post 10.

The same problem, the same shape of answer, one layer up. That is the most useful thing in this post.

Four: without the kernel

What changes. Everything above the driver, by choice.

The receive ring from Post 03 can be mapped into a process’s own address space. Once it is, that process polls the ring directly, reads frames out of it, and the kernel’s network stack never runs at all.

What the model predicts. You have not removed the work. You have taken ownership of it.

What you gain is exactly what the series spent posts on: no interrupt, no context transitions, no packet-struct allocation, no copy to userspace. Packet rates that are simply unreachable through the general-purpose path become routine.

What you must now provide is everything from Post 06 onward — demultiplexing, a TCP implementation, timers, retransmission, congestion control — plus the things the kernel was quietly doing that nobody lists, like responding to ARP. This is why kernel bypass belongs to load balancers, packet processing appliances and trading systems, and not to your API server.

There is also a middle path, and it is the thing Post 05 flagged and deferred. A small verified program can be attached to the driver and run on each frame before a packet struct is even allocated — early enough to inspect it, drop it, or redirect it, at a fraction of the cost of doing so in the stack. It is how modern denial-of-service filtering and software load balancing work: not by processing traffic faster, but by refusing it earlier, at the one point where refusing is nearly free.

Which is, once more, the same idea in new clothes. The whole series has been about where in the path a decision is made, and what it costs there.

What survived

Four substantial changes to the picture, and the same six ideas came through all of them intact.

Every boundary is still a queue, and container networking added several. Work still happens in contexts you do not control, though QUIC moved some of it into one you do. Buffers still have owners, which is why kernel TLS and kernel bypass are worth the trouble. Demultiplexing is still one motion repeated — QUIC just added another repetition, keyed on a connection ID. Backpressure still exists at every layer, though with QUIC you may now be the one implementing it. And framing was destroyed and rebuilt three times over on an encrypted HTTP/2 connection, once by TCP, once by TLS, once by HTTP/2 itself.

What changed in every case was where the lines are drawn. Never what happens at them.

What this series was actually for

Not the counters. You will forget most of them, and they are one search away.

The point was to make the middle of the machine legible. Before, there were two boxes: the internet, and your server. Now there is a path with about a dozen stages, each with a name, a purpose, an execution context, a queue, and a way to observe it — and when something goes wrong, a place to look.

The most useful consequence is a change of prior. When a request is slow, the question is no longer “which of my functions is slow,” because you now know that your code is the last two percent of a long journey and that most of the interesting failure modes live below it. You know what to ask, and roughly where.

That is a smaller claim than knowing how the network stack works, and a much more useful one.

If you want to go further

Three places, in increasing order of commitment.

The packagecloud posts on receiving and sending data are the definitive long-form treatment of Act II, with every counter and tunable in place. There is an illustrated version that is the better starting point.

For the physical layer, Ben Eater’s networking series does what no amount of prose can: puts an oscilloscope on a cable and decodes the signal.

And for anything specific, the kernel’s own documentation is far more readable than its reputation suggests — now that you know the vocabulary, which was always the barrier.

Thank you for reading. Go and look at ss -ti on something.