One Box

Courtyard lives on one rented machine. The chat server, the feed server, the database and a directory full of photos all sit on one box with one public address, which is the arrangement Part II Chapter 3 said you would have to make. A few thousand people are using it, and it is getting slow.

It is not broken. The tick after a message takes a beat longer than it used to, and photos in the feed take a moment to appear. Nobody has complained yet, but the trend is there to see, and you know from Part II that “slow” is three different things. Before you can find out which of the three you have, you need to know what has run out, and a machine can run out of several things, of which the processor is only the most obvious.

The first thing to do is to measure rather than guess. You point a load generator at Courtyard, a program that pretends to be a thousand users, then two thousand, then four thousand, and as the load climbs you watch two numbers: how many requests a second the machine completes, and how long each one takes.

The shape you see is one worth learning to recognise. At low load, throughput rises in step with demand and latency stays flat. The machine is keeping up, and each request costs the same whether there are ten a second or a hundred. Then, at some load, throughput stops rising and settles onto a plateau, and latency, which had been flat, begins to climb, gently at first and then steeply. Push further still and requests begin to fail outright, with timeouts and errors, as the loop from Part I Chapter 15 falls behind.

That plateau is the ceiling. The point where latency starts to climb is the knee, and it is the number you care about, because it marks the load at which the machine passes from comfortable to saturated. Everything past the knee is a queue forming, requests waiting for a resource that is fully busy, and Part II Chapter 5 told you what a queue does to latency. The knee is where a buffer begins converting your throughput problem into a delay problem. So Courtyard has a ceiling, as every machine does, and the question is which resource is the one that is full.

A machine can run out of four things, and each leaves a different signature. The first is the processor. Every request costs some computation: parsing, checking the security tag from Chapter 21, looking things up, building the response. Add requests and the computation adds up until the processor is busy every microsecond. The signature is processor utilisation at or near a hundred percent, with latency climbing smoothly as requests queue for the cores.

The second is memory. Each connection holds buffers, each request holds working data, and the database holds indexes it would like to keep in memory. Run out and the machine begins using the disk as pretend memory, which is catastrophically slower. The signature is memory full, then the disk suddenly busy for no visible reason, then everything crawling at once.

The third is the disk. Reads and writes to storage, whether photos, database records or logs, are served at a fixed number of operations a second. The signature is high disk utilisation, with the requests that touch the disk going slow while the ones that do not stay fast. That asymmetry is the tell.

The fourth is the network. The card and the wire have a rate, as Part I Chapter 23 established. The signature is bytes per second at the link’s limit, and latency that is fine until the moment it is not.

You look at Courtyard’s gauges. The processor is at ninety-eight percent. Memory is fine, the disk is idle most of the time, and the network card is barely awake. It is the processor. Every message, every feed request, every security handshake from Part I Chapter 21 costs a little computation, and a few thousand people’s worth of little adds up to all of it.

Naming the resource before changing anything is the discipline for the whole of this part, and it is the discipline most people skip. A fix aimed at the wrong resource does nothing, or makes things worse, and you do not find out until the next ceiling, which is now closer than it was, because you spent your effort in the wrong place.

The fix, once you know it is the processor, is to buy a bigger machine. That is the whole of it, and it is worth saying plainly because so much writing about systems treats it as something to be ashamed of, and it is not. The hosting company rents machines with twice the processors, and four times, and sixteen. You pick one and move Courtyard onto it, with the same code and the same configuration and one address change in the name record from Part II Chapter 4, and the ceiling goes up. It does not go up by the full factor you paid for, because some work does not parallelise, but it goes up substantially and immediately, and you have not changed a line of anything.

What you have done is what the field calls vertical scaling, or scaling up, and it has properties that nothing else in this part will have again. It is simple, in that one machine is still one machine and everything you learned in Part I about one machine still applies without modification. It is cheap in engineering time, because the change is a purchase rather than a redesign. And it is safe, because there is no new failure mode where there is no new component. The machine is bigger, and that is all that has happened. You should do this every time it is available, and you should do it before anything cleverer, because the clever things all cost you the simplicity you have right now.

Now do the arithmetic that tells you how long this lasts. Plot the machines on offer against two axes, what they cost and what they can do. At the low end the curve is friendly: twice the processors for roughly twice the money. Move up and it bends. The machine with eight times the processors costs more than eight times as much, because big machines are made in smaller numbers, and the one with sixty-four times costs a great deal more than sixty-four times. At the top there is a largest machine you can rent at any price, and past it the curve stops. You cannot buy a machine that does not exist.

Three other things erode the strategy before you reach the wall. The first is that some work does not spread across processors. Courtyard’s chat server is a single event loop, from Part I Chapters 9 and 15, and one loop runs on one core. A machine with sixty-four cores gives that loop one of them, and the other sixty-three sit idle unless you run sixty-four loops, which is, if you look at it squarely, sixty-four servers on one machine. You have begun building many machines inside one box, and the techniques of later chapters are already showing up.

The second is that one machine is still one failure. Part II Chapter 7’s lesson was that the box will not heal your own machine’s death, and a bigger machine is not a more reliable one. When the big machine has a bad day, everything is down, and a deploy that requires a restart is downtime for everyone, every time.

The third is that the price is paid continuously. A machine four times the size costs four times as much every month whether or not it is busy. At the low end that is trivial. At the high end you are paying a great deal for capacity you need only at the evening peak.

None of these stop you scaling up. What they do is tell you where it stops being the right answer, and the honest position is that for most systems, most of the time, it remains the right answer for far longer than people expect. Courtyard will get several more chapters of growth out of this one purchase. Buy the machine, and then start planning for the wall.

As for what can now fail that could not before, the answer is nothing, and that is the point of the chapter. You have bought capacity without buying complexity, and Courtyard’s failure modes are what they were yesterday. In the design document this becomes revision one: the processor ran out, you bought a machine with more of it, and there is no new failure. It is the last time that third field will be empty.