The Wire and the NIC

This post is about everything that happens before software is involved: how a continuous signal becomes discrete bits, how bits become a frame, and which decisions the card makes entirely on its own authority. It is the only post in the series with no kernel in it.

Bits are not on the wire

Start with the thing that is genuinely surprising: there are no bits on a wire. There is a voltage that varies continuously, and the receiving end has to decide, repeatedly, what it means.

To do that it needs to know when to look. If both ends had a perfectly shared clock this would be easy, but they do not — they are separate machines with separate crystals that drift relative to each other. So the receiver has to recover the sender’s clock from the signal itself, by watching for transitions and locking onto their rhythm.

Which creates a problem. A long run of identical bits produces no transitions, the receiver’s idea of the clock drifts, and it starts sampling in the wrong places. The naive encoding — high voltage means one, low voltage means zero — fails on any message containing a run of zeros, which is to say, all of them.

The fix is a line code: an agreed scheme that maps the bits you want to send onto signal patterns chosen to guarantee transitions. Fast Ethernet takes every four bits and sends them as a five-bit pattern drawn from a table where no entry has too few edges. Ten-gigabit optical links take sixty-four bits and send sixty-six, the extra two marking whether the block is data or control, with the payload scrambled so it cannot accidentally go quiet. Gigabit over copper is stranger still: it runs all four pairs at once, in both directions simultaneously, with each end subtracting its own outgoing signal from what it hears in order to recover the other’s.

None of which you need to remember. The part worth keeping is this: “the link is up” means two chips have agreed on a scheme and locked to each other’s timing. Everything above assumes that agreement and would be meaningless without it.

Where a frame begins and ends

The receiver now has a stream of bits and still no idea where anything starts.

So every frame opens with a preamble: seven bytes of alternating ones and zeros, which is the most transition-dense pattern available and gives the receiver’s clock recovery something to lock onto. Then a single byte whose last two bits break the pattern, marking the frame starts here.

After that comes the frame proper: six bytes of destination address, six of source, two saying what protocol is inside, then the payload, then a four-byte checksum covering everything.

And then, after the frame ends, twelve bytes’ worth of enforced silence before the next one may begin.

That silence is the last piece you need for the 941.

Deriving the number

Take a standard 1500-byte maximum transmission unit. One frame carrying a full-size packet costs, on the wire:

• 1500 bytes of IP packet
• 14 bytes of Ethernet header
• 4 bytes of checksum
• 8 bytes of preamble and start marker
• 12 bytes of interframe gap

1538 bytes, of which 1518 are the frame and 20 are ceremony.

A one-gigabit link is 1,000,000,000 bits per second, which is 125,000,000 bytes, which gives 81,274 frames per second.

But the 1500 bytes inside are not all yours either. Twenty go to the IP header and twenty more to the TCP header, leaving 1460 bytes of payload. And 81,274 frames each carrying 1460 bytes is 949 megabits per second.

Still not 941. The last twelve bytes are TCP’s timestamp option, which nearly every modern stack negotiates by default, and which reduces the payload to 1448. Redo it: 941 megabits per second.

What the card decides on its own

The frame is now inside the card. Before the CPU learns anything has happened, several decisions have already been made and are not revisited.

Is it intact? The card recomputes the checksum over the frame and compares. A mismatch means the frame was corrupted in transit — a marginal cable, a failing transceiver, electrical noise — and it is discarded immediately. A counter increments and nothing above ever learns the frame existed. This is why a steadily climbing error counter on one port is one of the most useful signals in networking: it localises a physical fault to a specific cable.

Is it for me? The card compares the destination address against the address it was assigned, against the broadcast address, and against a set of multicast groups the driver told it to accept. No match, and the frame is dropped — which is what keeps a machine on a busy segment from being interrupted by everyone else’s traffic. Putting the card in promiscuous mode is exactly the act of turning this filter off, which is why running a packet capture can measurably increase a machine’s load.

Are the upper-layer checksums right? Most cards will, as a favour, verify the IP and TCP checksums too and record the result in the descriptor, so the kernel does not have to walk the bytes again.

Which core should handle this? This is the surprising one. A modern card has many receive rings, not one. It hashes the packet’s source and destination addresses and ports, and uses the result to pick which ring — and therefore, since each ring has its own interrupt, which core — will process this packet.

Sit with that for a second. A network card is making a CPU scheduling decision, in hardware, before the operating system knows a packet arrived. It is a good decision, because hashing the connection identity means every packet of a given connection lands on the same core, which keeps that connection’s state in that core’s cache. But it is being made by a device, on the basis of bytes that came from the network, and the kernel’s scheduler has no say in it.

When the wire is not a wire

The series is called From Wi-Fi to Route Handler, so this is the place to be precise about where Wi-Fi actually enters.

802.11 is a considerably harder problem than Ethernet. The medium is shared and nobody can hear collisions while transmitting, so every unicast frame must be individually acknowledged and retried if it is not. The data rate is negotiated continuously against measured conditions and can change frame to frame. There is a power-saving protocol under which the access point buffers frames for sleeping clients. And the frame format carries up to four addresses rather than two, because a frame may be travelling between a client and an access point rather than simply between two endpoints.

Almost none of which reaches your server.

In the ordinary arrangement — wireless client, wired server — the access point is the boundary. It receives the 802.11 frame, handles the acknowledgements and retries, strips the wireless framing, builds an ordinary Ethernet frame, and puts it on the wire. What arrives at your machine has no trace of Wi-Fi in it. The retries happened; the latency variance they cause is real and will show up later as jitter; but the frame is a frame.

When the server itself is wireless, the translation happens on the server instead. The driver and the kernel’s 802.11 layer handle the wireless machinery, convert the data frame into Ethernet form, and hand it to exactly the same function an Ethernet driver would have called. From that instruction onward, the two paths are the same code.

A packet that cannot exist

One last thing the card and driver do, which will confuse you at some point if nobody warns you.

Run a packet capture on a busy server with a 1500-byte MTU and you will see packets of 30,000 or 60,000 bytes. Not fragments. Single packets, reported as single packets, on a link physically incapable of carrying them.

They are real in the sense that the kernel really did process one structure of that size. They just never existed on the wire. Several consecutive segments of the same connection arrived back to back, and rather than walk each one separately through the entire stack — the routing lookup, the firewall, the socket lookup, the TCP state machine — the receive path merged them into a single large packet first, because all that per-packet work is identical for all of them.

The capture is taken after the merge, so the capture shows the merge.

And this is the perfect place to leave off, because the motive behind it is about to become the subject of the whole next post. Merging packets exists for one reason: the fixed cost of handling a packet is large enough that it is worth going to considerable trouble to handle fewer of them.

Hold that thought. It also explains the single cleverest thing in the receive path, which is how the kernel manages to take far fewer than one interrupt per packet.

See it for yourself

Every offload described above can be listed, and most can be turned off, which is occasionally the fastest way to find out whether one of them is lying to you.

Features for enp3s0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on

For the physical layer, ip -s link show gives you the error and dropped counts per interface, and ethtool -S gives you the card’s own far more detailed set. If the CRC error counter on one interface is climbing and its neighbours are clean, you are looking at a hardware fault on one cable, and no amount of tuning above it will help.

What to carry forward

The card hands the kernel a frame that is already known to be intact, already known to be addressed here, already checksum-verified at two layers, already assigned to a core, and possibly already merged with several of its successors. A great deal has been decided.

What it has not done is tell anyone. The frame is sitting in main memory, in a buffer from Post 03’s ring, and the CPU is still doing whatever it was doing.

Getting its attention — without doing so 812,000 times a second — is the next post, and it is the best idea in the stack.