When the Path Breaks

Somewhere in the Tasman Sea, a ship drags its anchor across the seabed and severs a cable the thickness of a garden hose. This happens more often than you would think, a couple of times a week, somewhere in the world.

In Sydney, Anaya’s chat stops. The tick does not appear. The online dots go grey and the feed spins. For about ninety seconds, Courtyard does not exist. Then it comes back, a little slower than before, and nobody did anything.

To see what happened, go back to Chapter 2 of this part. Each box in the middle knows which neighbour is closer to each prefix, and it learned that from gossip. The boxes at either end of the cut cable were neighbours, and now they cannot hear each other. Each notices, after a few seconds of silence, because silence is the only signal there is, and each withdraws the routes it had learned through that link. It tells its other neighbours that it can no longer reach these prefixes this way. Those neighbours update their tables and tell theirs, and the withdrawal ripples out across the world’s tables at the speed of gossip.

Meanwhile, other paths exist. There is a cable that goes the long way round, by way of another country, and the boxes along it have been announcing Mira’s prefix all along, only as a worse option. As the withdrawals arrive, boxes that used to prefer the cut cable switch to the long way. Piece by piece, table by table, the world reroutes. The field calls this convergence, and from Sydney it looks like this.

The first seconds are a black hole. Every box on the old path is still forwarding into the cut, because none of them knows yet, and every piece is dropped. Chapter 11’s timers fire and resend, into the same hole. The next minute or so is partial. The withdrawals have reached some boxes and not others. A piece might be forwarded by a box that has updated toward one that has not, and get sent back again, in a transient loop that the hop limit eventually kills. Some pieces get through on the long path and some do not, and Anaya’s connection flickers. Then it is over. Every table agrees, pieces flow by way of the other cable, and the round trip is thirty milliseconds longer because the path is longer. Chapter 6 told you what that costs, but it works.

Nobody fixed anything. The box healed itself, because its tables are gossip and gossip eventually reaches everyone. That is the box’s own resilience, and I find it one of the more remarkable facts in the subject: a cable on the seabed is cut and, ninety seconds later, without any human deciding anything, the world has routed around it.

For those ninety seconds, though, the contract was void. Not violated, because the box never promised delivery, but void in practice, since nothing got through. The “usually, eventually” of Chapter 1 has a failure mode, and this is it. It is measured in seconds to minutes, and there is nothing you can do to shorten it. What you can do about it comes down to two things, and both concern the timers of Part I.

The first is to know that this exists when you pick thresholds. The heartbeat of Chapter 13 declared Anaya dead after three missed pulses, which at a thirty-second interval is ninety seconds. She was not dead. The path was converging. If Courtyard’s server had taken some expensive action on that declaration, closing her session, telling her contacts she had gone offline, handing her connection to another machine, it would have acted on a false death caused by a router in another hemisphere catching up on gossip. Every failure detector you build sits on top of a box that goes briefly deaf, and thresholds should be set with that in mind.

The second is to let the transport do its job. Anaya’s connection did not need to be torn down and rebuilt. The retransmission of Chapter 11, with its backing-off timer, kept trying, and when the path came back, the next retry got through and the connection resumed where it had left off. A client that gives up and reconnects at the first timeout pays the full handshake bill of Chapter 6 on a path that is about to recover anyway. Patience, in the form of a timer that backs off but does not quit, is usually the right response to a break you cannot see.

Now consider a different kind of failure. The cable is fine, but the hosting company that runs the machine Courtyard’s server lives on has a power problem, and the machine goes down. Every user everywhere, in Sydney, in London, in the courtyard, is dark.

This time the box does nothing. It routes around its own failures; a machine at the edge is not its problem. Mira’s prefix is still announced, the tables are all correct, and pieces arrive at the building perfectly and are dropped by a dead machine. The box has kept every clause of its contract. Courtyard has one server, and one is a number that fails.

The obvious remedy is a second server, on a second host, in a second building. Two machines, two addresses. Point the name at both, since the lever of Chapter 4 allows a name to resolve to several addresses. Clients pick one, and if it does not answer they try the other.

That works, and it is slow in two ways you can now name. The client has to discover that the first address is dead before trying the second, which means a timeout, from Chapter 11 or Chapter 13, of seconds at least. And the dead address stays in the name record until Mira removes it and the world’s cached copies expire, which by Chapter 4’s lifetimes is minutes to hours. Users get a working Courtyard eventually, after each of them has individually timed out on a corpse. There is a better trick, and it uses the box itself.

Recall precisely what the box does with a prefix. It forwards toward whichever neighbour announced it, preferring the one it considers closer. Now suppose Mira has two servers, one in London and one in Sydney, and both of them announce the same prefix. Not two prefixes and two addresses, but the same address, announced into the box from two places at once.

The box does not know or care that there are two machines. It sees the prefix announced by two neighbours and does what it always does: each box picks the announcement it considers closer. Boxes near London pick London’s. Boxes near Sydney pick Sydney’s. Anaya’s pieces, addressed to the one address 203.0.113.50, arrive at the Sydney machine, because from every box along her path, Sydney’s announcement was the nearer one. Mira’s own pieces, to the same address, arrive in London. One address, two machines, and it was the box that chose which. Not a name lookup, not the client, not anything Mira runs.

Now kill the Sydney server. It stops announcing. The withdrawal spreads, in the same convergence as the cable cut, taking seconds to a minute, and every box that preferred Sydney’s announcement falls back to London’s. Anaya’s pieces, still addressed to 203.0.113.50, now arrive in London. She noticed a hiccup. She did not time out on a dead address, did not wait for a name record to expire, and did not do anything at all. The box rerouted her to the surviving machine because, as far as the box was concerned, one of the two paths to that address had gone away.

The field calls this anycast, and it is the first time in this book that one address has meant many machines. It gives you three things at once. There is failover, with no client involvement, at the speed of convergence. There is proximity, with every user reaching the nearest copy and no geographic lookup, because the box’s own notion of closer did the work. And it is invisible: to every client there is one Courtyard, at one address.

It is not free, and the costs are the ones the tower predicts. Long conversations can be torn. Anaya has a chat connection open to the Sydney machine: a four-tuple in that machine’s table, with the sequence numbers of Chapter 12 and the window of Chapter 14. The box converges and starts sending her pieces to London. London has never heard of that four-tuple, so it sends a reset, and her connection dies and has to be rebuilt. Anycast is perfect for short exchanges, one question and one answer, like the name lookups of Chapter 4. It is workable for longer ones only if the tables are stable, which in practice they mostly are. Where they are not, the answer is to anycast to a front door that then hands the connection to a specific machine and keeps it there, and that is a chapter of Part III.

Convergence is still the gap. Failover takes as long as the withdrawal takes to spread. That is faster than a client timeout plus a name expiry, much faster, but it is not zero. And you have to be able to announce at all. Putting an announcement into the box means being a participant in the gossip from the interlude, or, more realistically, renting the ability from a provider who is. It is a lever, but it is a lever you reach through someone else.

You have watched the box break and heal on its own, and you know what the healing looks like, how long it takes, and what your timers should assume about it. You have seen that the box will not heal your failures, and you have built the trick that survives them, and the trick turned out to be something bigger than failover. Consider what you did without meaning to. You put Courtyard in two places. You made one address mean two machines. You had users in different hemispheres reach different copies, and you made the copies cover for each other when one died. You have built, in miniature, the shape of every large system in the world, and you did it with a router’s table and a duplicated announcement.

Part II asked what the box owes you. The answer was two promises, eight refusals, and one clause that made you build a server. Then it gave you the levers: a name, a lifetime, an announcement. And it showed you the enemy, which is distance, and which cannot be fought but only avoided. Part III is about avoiding it, and about what happens when strangers arrive, many strangers, and Courtyard’s one machine, then its two, then its many, run out of something, over and over, each time in a new way.