Nobody Remembers Numbers

Courtyard’s server lives at 203.0.113.50, and you have told everyone so. Then the hosting company has a problem with that machine, and Mira moves the server to a new one at 198.51.100.7. Every client in the world has 203.0.113.50 baked in, and every one of them is now talking to a machine that no longer runs Courtyard. Mira has moved house and nobody has her new address, and she has no way to tell them, because the only way of telling them is through the thing that just broke. Underneath that there is the smaller problem, which was always there: nobody could remember 203.0.113.50 in the first place.

The fix is obvious in outline. Give the server a name, courtyard.example, and have clients use the name. Somewhere, keep a table from names to addresses. When Mira moves the server she changes one row in the table, and every client, the next time it looks up the name, gets the new address. The name is stable, the address underneath it is not, and the table is the seam between them. This is Chapter 6’s two-names idea, one floor up. The name says who, the address says where, and you need a way to go from one to the other. The whole question is where the table lives.

The first thing to try is giving every machine a copy. Each machine has a file with every name and address in it, and lookups are instant and local. For a courtyard that is fine. For the world it is Chapter 2’s map all over again: the table changes constantly, every change has to reach every copy on every machine, and it never will, because copies go stale faster than they can be refreshed. When Mira moves the server she would have to push a new file to every device on Earth. This is in fact how it worked in the early days, one file, maintained by one office, downloaded by every machine, and it stopped working when the number of machines passed a few hundred.

The second thing to try is one machine that everyone asks. Put the table on a single server, and changes are instant, because there is one copy. But one machine cannot answer every lookup on Earth. It is a single point of failure, and when it is down nothing has an address. It is far away from almost everyone. And one organisation would own every name, which nobody would agree to.

The third thing to try is hierarchy, which you have now done twice, so do it again. Split the namespace by suffix and delegate each piece to a different authority. At the top, the root knows one thing: who is responsible for each top-level suffix, .example, .com, .org. It knows no names below that; it knows whom to ask. The authority for .example knows who is responsible for each name directly under it, so it knows that courtyard.example is Mira’s, and it knows no names below that. Mira runs, or rents, a server that answers for everything under courtyard.example: chat.courtyard.example, feed.courtyard.example, and whatever else she likes.

To resolve chat.courtyard.example, you ask three questions of three different machines:

you → root:                who handles .example?
root:                      ask 192.0.2.1
you → 192.0.2.1:           who handles courtyard.example?
192.0.2.1:                 ask 198.51.100.53   (Mira's name server)
you → 198.51.100.53:       what is chat.courtyard.example?
198.51.100.53:             198.51.100.7

Each authority holds a table sized to its level. The root has a few hundred rows, .example has millions, and Mira has a dozen. No one holds the whole thing. Mira controls her names without asking anyone, and when she moves the server she changes a row on her own server, and the next lookup gets the new address.

This is Chapter 5’s move, to impose a prefix structure so that the middle knows about regions instead of individuals, applied to names read from right to left instead of addresses read from left to right. What you have built is the Domain Name System, DNS. The machine that answers for a piece of the tree is authoritative for it, and handing a piece of the tree to someone else is delegation.

Three round trips per lookup is a lot to pay before you have sent a single byte to Courtyard, and every machine on Earth asking the root for every lookup would crush it. So most machines do not walk the tree themselves. They ask a resolver, a machine run by their provider or a public one, and the resolver walks the tree on their behalf. It asks the three questions, hands back the answer, and, most importantly, remembers it. That is a cache, and you know from Chapter 6 what comes next.

The resolver holds chat.courtyard.example → 198.51.100.7. The next person who asks gets the answer at once, from memory, with no round trips at all. That is fine until Mira moves the server again.

Chapter 6 asked how long to trust a cached answer, and found there was no right number, only a trade. DNS makes one addition to that: the owner of the record chooses. Alongside every answer, Mira’s server sends a lifetime, a number of seconds for which the resolver may keep it. She is saying that she promises not to change this for at least this long, and that the resolver may remember it for up to that long. The lifetime is called the time to live, or TTL, and it is the dial that governs everything that follows.

Set it long, a day say, and lookups are fast and Mira’s server is barely bothered. But when she moves, every resolver that cached the old address keeps handing it out until its copy expires, which could be as much as a day later. Different resolvers cached at different moments, so they expire at different moments. For a day, some of the world reaches the new server and some of the world reaches the old one, and which is which depends on when each resolver last asked. Mira has moved, and the world finds out one resolver at a time, on a schedule she set a day ago.

Set it short, a minute say, and the world follows changes within a minute. But now every resolver asks Mira’s server again every minute, for every name, forever. That is load on her server, and a round trip added to many of her users’ first requests.

This is the story every operator eventually lives, and it goes: we changed DNS, and half the internet is still hitting the old box. People call it “propagation”, which is the wrong word, because nothing propagates and nothing is pushed anywhere. Copies expire, each on its own clock, and the new answer is fetched only when an old one runs out. The world does not learn that Mira moved. It forgets where she was.

Because that is so, there is a playbook. Days before the move, while the address is still the old one, Mira lowers the TTL from a day to a minute. She waits a day, so that every resolver’s day-long copy has expired and been replaced with a minute-long one. Then she moves, and within a minute the world follows. Then she raises the TTL again. The lowering has to happen a full old TTL in advance, because until then resolvers are still holding the old lifetime along with the old address. If you have ever wondered why a careful migration starts with a DNS change a week before anything visibly happens, that is why.

Step back and look at what you have built, because it is a shape you will see for the rest of the book. There is one true copy of each record, on the authoritative server, controlled by its owner. There are thousands of cached copies scattered across resolvers, each allowed to be stale for a bounded time, each expiring on its own schedule. Readers almost always read a copy and not the truth. The owner controls how stale the copies may be, and pays for freshness with load. Nobody coordinates the copies. They expire, and that is all.

That is a distributed database, the first you have built, and a purer example than most. Everything about it recurs in Part III. When Mira adds a second copy of Courtyard’s database so that reads do not overwhelm the first, the copy will lag behind the truth, and she will be choosing how much lag is acceptable, and users will read the copy without knowing it, and you will recognise the TTL map. When she puts copies of a video in a hundred cities, each with a lifetime, and then updates the video, and some cities show the old one for a while, you will recognise it again. Staleness by design, owner-chosen, bounded but not zero: it started here.

Three details will bite you later. The first is that absence is cached too. You mistype a name; the resolver walks the tree; Mira’s server says there is no such name. That answer is cached, for a lifetime Mira also controls, so that a flood of lookups for a name that does not exist does not reach her server every time. Then you create the name. The resolver, asked again, says from memory that it does not exist, and goes on saying so until the negative copy expires. A brand-new name can be invisible from some resolvers for minutes after it is created, for the same reason that a moved one is stale.

The second is that a lookup is a datagram. One question, one answer, and no ongoing conversation, which is Chapter 17’s third case. DNS runs on the datagram transport by default because a handshake would cost more than the exchange. The question fits in one piece, and so does the answer.

The third is that the lookup is in the clear. Those questions cross the box unencrypted, and anyone on the path can read them. Chapter 21 warned that a secure channel hides the content but not the metadata, and the name you looked up is metadata. It leaks here, to your provider and to everyone in between, even when everything that follows is https. Encrypting the lookup itself is possible and increasingly done, but it is not the default.

One last thing, and it is the reason this chapter is longer than the mechanism alone would need. DNS is the most important lever you have on the box. It is not a clever thing; it is a cached table. Its importance lies in what it lets you do without touching a single client. To move a server, change a record. To replace a machine, change a record. Point a name at several addresses and let clients pick among them, which is the crudest possible way to spread load across machines and is Part III’s first move. Point a name at different addresses for people in different places. Take a machine out of service by removing it from the record and waiting one TTL.

Every one of Part III’s changes to Courtyard, every new machine, every second copy, every front door, every region, begins with a DNS change, and every one of them is bounded in speed by a TTL that someone chose. When you are wondering how long a change will take to reach users, the answer is almost always however long you told the world it could remember the old one.

Courtyard has a name now. The name is stable and the address underneath it can change, and when it does, the world follows on a schedule Mira set. You have built a distributed database without meaning to, met staleness in its clearest form, and picked up the lever you will be pulling for the rest of the book. Next, everything works, and it feels slow, and more bandwidth does not help.