Compute and State Are Different Animals

The bigger machine bought Courtyard a good stretch, and then people discovered the feed.

Photos are popular in a way that messages are not. One photo gets looked at hundreds of times, and each look is a read from the disk. On a busy evening the disk serves photo after photo, flat out, with the utilisation gauge from last chapter pinned at the top. And chat, which was fine, is suddenly not. The tick after a message takes a second, sometimes two. Chat does not touch photos, so it is fair to ask why it should be slow.

It is slow because a chat message, to be stored, is a write to the database, and the database lives on the same disk that is serving the photos. The database’s write joins the queue behind a few hundred photo reads and waits its turn. The chat server waits for the database, and you wait for the tick. Nothing about chat has changed. The feed got popular, the two share a resource, and so the feed’s success is chat’s slowdown. The field calls this contention, and the general lesson is that two things on one machine are never really independent. They are coupled through whatever resource they both touch, and the coupling shows up only under load, and only in the direction you were not watching.

The disk is not the only coupling on that box. Deploying a new version of the chat server means restarting a process on the machine that also runs the database. A bad version that leaks memory starves the database of memory. A bad version that logs too enthusiastically fills the disk, and a full disk stops the database from writing anything, and a database that cannot write is Courtyard, gone, on account of a logging bug in the chat server.

And if the machine itself dies, through power or hardware or an operator’s mistake, the chat server, the feed server, the database and every photo die together. The servers can be restarted from a fresh copy in a minute. The database and the photos are the data. If the disk is gone, Courtyard is not slow; it is over.

The move is to stop sharing. You set up three machines. One runs the application, meaning the chat and feed servers; one runs the database; one holds the photos. They talk to each other over the network. The contention disappears at once. Photo reads hit the photo machine’s disk, database writes hit the database machine’s disk, and neither queues behind the other. A chat-server deploy restarts a process on a machine that holds nothing precious. A logging bug fills a disk that the database is not on. And each machine can be sized for its own job. These are Chapter 1’s four gauges again, except that now you buy the right one for each: processors for the application, memory and fast disk for the database, and sheer disk capacity for the photos.

That is the mechanical benefit, and it is real. But the reason the chapter matters is what you see once the three are apart. They are not three workloads of the same kind that happen to be on different machines. They are three different kinds of thing.

The application is disposable. The chat and feed servers hold nothing between requests, which was Part I Chapter 20’s choice, made deliberately and paid for here. Kill the process and start it again and nothing is lost; the next request arrives complete and is answered. Run two copies and either can answer anything. Replace the machine with a different one and nobody notices. The application is interchangeable, and interchangeable things are easy to restart, easy to duplicate, and easy to reason about, because there is nothing inside them that matters.

The database is precious. It holds the truth: every message, every account, every like. It is one machine, and it must be that machine, because the truth lives there and nowhere else. You restart it carefully, at three in the morning, with a plan. You back it up, because if this disk dies Courtyard dies with it. You cannot casually run two of it, since two copies of the truth are two truths, and the next chapter is about what that costs. Everything hard in Part III is about this box.

The photos are huge and immutable. A photo is uploaded once and never edited. It is created, viewed ten thousand times, perhaps deleted, and never changed. There are a great many of them and each is large, and together they dwarf the database a thousandfold. But they do not need what the database needs. A photo that is a second stale is not a thing that can happen, because photos do not change, and any copy of a photo is as good as any other, forever. What they need is capacity and read throughput, and they get Part I Chapter 20’s cacheability for free: a GET for photo 42 today returns what it will return next year.

This asymmetry organises the rest of the part, and it can be put in a sentence: state is the hard part, and everything else is easy. Moving computation is trivial; it is a process, and you can start it anywhere. Moving state, which means deciding where the truth lives, keeping copies in agreement, and surviving the loss of the machine that held it, is the whole of distributed systems. When a chapter in this part is hard, it will be because it is about the database. When a chapter is easy, it will be because it is about the application. The photos turn out to be a third story of their own, easier than the database because nothing changes, and harder than the application because there is so much of it.

Now for the bill, which is the first real one. Yesterday the chat server talked to the database by calling a function on the same machine, which was instant and never failed. Today it talks to the database over the network, through a socket, across a wire, possibly through a switch, to another machine. The whole of Part I’s tower now sits between two pieces of Courtyard, and Part II’s contract applies to it. The request can be delayed, the response can be lost, and the database machine can be up while the network to it is not.

So the chat server needs things it did not need yesterday. It needs a timeout on every database call, some length of time after which it decides the database is not going to answer. That is Part I Chapter 11’s question, which has no right answer and a wrong one in either direction: too short and you fail requests the database was about to complete; too long and a slow database freezes the event loop’s handler for the duration, which Part I Chapter 15 told you must never happen. It needs a decision about what to do when the database does not answer. Fail the request, or retry? And if you retry, is the write idempotent, or did the first one land, so that the retry stores the message twice? That is Part I Chapter 20, now inside Courtyard. And it needs a way to tell which machine is the problem when a user says “it’s slow,” because yesterday there was one machine to look at and today there are three, and a slow feed could be the application, the database, the photo store, or the network between any pair of them.

It is a small bill today. Three machines, one building, a switch between them, a round trip of a fraction of a millisecond. Timeouts will be generous and will rarely fire. But the kind of bill is new. Courtyard now contains a network, and a network is the thing Part II spent seven chapters describing. Every split from here on will make this bill larger, and Chapter 13 will make it enormous. Revision two of the design document reads accordingly: the disk ran out; the application, the database and the photos were separated onto three machines; and the network is now inside Courtyard, so that calls between components can be slow, lost, or half-complete.