A Network of Networks

Open the box by one inch. You agreed that you would not need to know how it is built, and you are not going to look at that. What you want to see is why it keeps the contract it keeps and not a better one.

If you were designing the box from scratch, the first thing you would want is a map: every machine inside, every link between them, so that when a piece arrives for Mira you can plot the route and send it along. Nobody has such a map. That is not because nobody thought of it but because it cannot exist. Consider what the map would have to contain. Hundreds of thousands of boxes and millions of links, owned by tens of thousands of separate organisations who do not share their internals and do not want to. Links go down and come back constantly, somewhere in the world, every second: a cable is cut, a power supply fails, a router reboots for an upgrade. A complete map would be wrong the moment it was drawn. To keep it right, every change anywhere would have to be reported to every box everywhere, immediately, which amounts to a message to every box in the world every second. This is Chapter 6’s stale cache at planetary scale, a copy of everything, going stale faster than it can be refreshed.

So there is no map, and yet your piece reaches Mira. The way it does so is that each box knows which of its neighbours is closer to each prefix, and that is all it knows.

Go back to Chapter 5’s router. It had a card on each side and a small table that said, for this prefix, use that card. It did not know the whole path to any building; it knew which direction each building lay in. Now scale that up. A box in the middle of the city has a dozen neighbours instead of two, and a table with hundreds of thousands of rows instead of two, one row per prefix, but each row still says the same kind of thing: for this prefix, hand the piece to that neighbour. Not the path, only the next step.

A piece for Mira arrives. The box looks up her prefix, finds the neighbour, hands the piece over, and forgets it. That neighbour does the same, and so does the next. No box knows where the piece has been or where it will go after the next hop. Each makes one local decision, based on one local table, and the piece arrives because every table, individually, points a little closer.

What you have just watched is called hop-by-hop forwarding, and what makes it interesting is that it produces a global result, pieces crossing the world, out of purely local rules. There is no coordinator and no plan. Every box knows the next step, and the next steps add up.

Hold that picture and walk back through the refusals of the last chapter, because each of them falls out of it.

Take order first. Your piece 1042 arrives at a box, and the table says neighbour B. But the box has two equally good neighbours for that prefix and spreads traffic between them, or the table was updated between the two pieces because a link somewhere changed. Piece 1043 goes to neighbour C. Different neighbours mean different paths and different lengths, and 1043 arrives first. Nobody decided to reorder them. Two local decisions happened to differ.

Take delivery. A piece arrives at a box whose queue toward the next neighbour is full. The box has nowhere to put it, so the piece is dropped. Or a link has just failed and the table has not caught up, so the box hands the piece to a neighbour, and the neighbour hands it back, each box’s table saying the other is closer, and the piece circles. That raises a question of its own, which is how such circling ever ends.

It ends because every piece carries a counter, set by the sender, and every box that forwards the piece subtracts one from it. When the counter reaches zero, the box discards the piece instead of forwarding it. The counter is called a hop limit. A piece caught in a loop between two boxes with inconsistent tables goes round until its counter runs out and then vanishes. That is a loss, and it is a loss by design, because a piece that could circle forever would eventually fill the box with immortal traffic. The hop limit is the box’s guarantee that every piece, delivered or not, is eventually gone.

Take timing. Path length varies with which neighbours were chosen. Queue depth varies with what everyone else is sending through each box at that instant. Two pieces a millisecond apart can cross different numbers of boxes and wait behind different amounts of traffic. There is no fixed delay because there is no fixed path.

Take capacity. Each box knows how full its own queues are. No box knows how full the path is, because no box knows the path. The only thing a full box can do is drop, and the drop is the only signal that reaches the sender, which is why Chapter 16 had to probe.

Take the occasional duplicate. A transient loop can, in rare conditions, deliver a piece and also send a copy round again before the tables settle. Combine that with the ends’ retransmissions and you get the occasional piece that arrives twice.

Every refusal is the same refusal seen from a different angle. The box makes local decisions with local knowledge, and local decisions do not add up to a promise.

Open the box one more inch, because what is behind it sets up the last chapter of this part. If no box has the map, how does a box know that neighbour B is closer to Mira’s prefix? It was told. Each box announces to its neighbours the prefixes it can reach; the neighbours pass that on, adding a step; and the rumour spreads outward from every network until every box has heard, from some neighbour, a way toward every prefix. The tables are built from gossip.

That means the tables are copies of a rumour, and copies go stale. When a link fails, the box that owned it stops announcing the prefixes it reached that way. Its neighbours notice, update their tables, and stop announcing those prefixes to their own neighbours, and the news ripples outward. But rippling takes time, seconds and sometimes minutes, and during that time some boxes have heard and some have not. The ones that have not are still forwarding toward the dead link. Pieces are dropped, or loop, or take a long way round. The contract is not being broken, since it never promised delivery, but it is being honoured at its worst.

The interval while the rumour spreads and the tables disagree is called convergence, and it is the one honest gap in the whole scheme. You will watch it happen in Chapter 7 of this part, and you will see the trick that makes it not matter for the things you build.

What you have learned is that the box has no map, that each machine in it knows only a next step, and that the tables are gossip and go briefly stale when things change. From that, every clause of the contract follows, and a reordered piece or an unexplained loss will never surprise you again, because you know what kind of thing produced it.

What you have not learned is how the tables are computed, how boxes choose between two neighbours, what “closer” even means when the neighbours belong to different companies, or how any of the machines inside are built. Those are real subjects and none of them are yours. A person who builds things on the box needs to know what it promises and why the promise is shaped that way, and you now know both. Close the box.

The contract makes sense now. Its refusals are neither arbitrary nor lazy; they are what you get when forwarding is local and the map is impossible, and the alternative would be a middle too complicated to grow.

There is a half-chapter next, and it is optional. It concerns that word “closer”, and the fact that inside the box the neighbours are companies, and closer is negotiated. You do not need it to build anything. You need it to understand the news. After it comes Chapter 3, which is not optional at all, because it is the clause you have not hit.