Interlude: Routing Is Politics

One morning a headline says that half the internet went dark for two hours because of a mistake made in a country you have never visited, and somewhere in the article is the word BGP. This half-chapter is about what that sentence means. You may skip it, because nothing in it is a lever you can pull. It is here so that the news makes sense.

Chapter 2 said that each box hands a piece to the neighbour that is closer to the destination. Inside one organisation’s network, closer means what you would expect: fewer hops, less delay, a wider link. The network’s own engineers set the tables. But the box is not one network. It is tens of thousands of them, your internet provider, Mira’s, the university across town, the companies that run the great cables under the ocean, the ones that host Courtyard’s server. Each of them is a business. Between them, the tables are not set by engineers optimising for delay. They are set by contracts.

When two networks connect, the arrangement is one of two kinds. In the first, a smaller network pays a larger one to carry its traffic to anywhere at all. Your provider pays a bigger provider; the bigger provider pays, or is, one of the handful of networks that reach everything. Money flows up and reachability flows down. The larger network announces to the smaller every prefix in the world, because it is being paid to carry traffic toward all of them. This arrangement is called transit.

In the second, two networks of similar size agree to exchange traffic between their own customers, for free, because that is cheaper than both of them paying a third party to carry it. Your provider and Mira’s, if they are both large enough and both present in the same city, might make such an arrangement. Each announces to the other only its own customers’ prefixes, and not the whole world, because it is not being paid to carry the other’s traffic anywhere else. This is called peering.

So what a box tells its neighbour it can reach, the gossip of Chapter 2, is filtered by what it is willing to carry for whom. And when several neighbours can all reach a prefix, the one a box chooses as closer is whichever costs it least: a customer who pays, before a peer who is free, before a provider who charges. The shortest path does not enter into it unless everything else is equal. The most common policy has the technical name hot-potato routing. Get the piece off your network and onto someone else’s as fast as possible, because every hop inside your network costs you and every hop inside theirs costs them.

Consider what that does to your pieces. Mira lives ten kilometres from you. Her provider and yours do not peer in your city; they peer in a city three hundred kilometres away, because that is where they both happen to have equipment. Every piece between the two of you goes there and back. Six hundred kilometres to cross a city, because a contract between two companies says so.

Latency, in other words, is partly a business outcome. When Chapter 6 of this part tells you that the speed of light is the floor, remember that the path the light takes was negotiated.

Now the part that becomes news. The tables are built from what neighbours announce, and for most of the internet’s history, and still mostly today, announcements are believed. A network says that it can reach a prefix, and its neighbours update their tables and pass the news on. There is no central registry for a box to check against, and no proof of ownership attached.

So a network can announce a prefix it does not own. It can do so by mistake, through a typo in a configuration or a filter that should have been there and was not, or it can do so on purpose. Its neighbours believe it. Their neighbours believe them. Within minutes some fraction of the world’s boxes hold a table entry saying that the prefix is over here, and traffic for it flows to a network that has no idea what to do with it, or that knows precisely what to do with it and should not.

In 2008 a provider in Pakistan was ordered to block a video site within the country. Its engineers did so by announcing a more specific prefix for the site’s addresses, so that traffic inside the country would go to a black hole. This was Chapter 5’s hierarchy turned into a weapon, since a longer prefix wins. The announcement leaked past their border. For about two hours a large part of the world’s traffic to that video site went to Pakistan and vanished. Nobody attacked anything. Someone’s gossip was wrong, and everyone’s copies believed it.

This is called a route leak, or when it is deliberate, a hijack, and there have been many of both. The fix, a cryptographic proof that a network is allowed to announce a prefix, checked before the announcement is believed, exists and is being deployed, slowly, because it requires tens of thousands of independent businesses to each do work that mostly benefits others.

You will never configure any of this. Your provider’s contracts, their peering and their filters are none of them yours, and none of them should be. What you take from this interlude is the vocabulary. When a headline says BGP, it means the gossip between networks. When it says hijack or leak, it means a wrong announcement was believed. When it says convergence, it means the tables were catching up. And when a service you build is unreachable from one country and fine from another, this is one of the things it might be. It will not be your fault, and there will be nothing you can do about it except wait and post a status update. Close the box, for good this time.