One Front Door

The database is taken care of, for several chapters at least, and the trouble moves forward to stand in front of it. The application servers are the chat and feed servers from Chapter 2, the machines that hold nothing and do the computation, and one of them, however large you make it, has the ceiling that Chapter 1 found. Courtyard has now passed that ceiling. You need several, and the first thing you discover is that there is nothing hard about running several. These servers are stateless, the disposable animal of Chapter 2, so you install the same code on five machines and start it.

Then you look for the difficulty and find it in the addresses. Each of the five machines has an address of its own. Courtyard has one name. A client looks up that name, gets one address, and opens one connection to it. You have five servers and a client can find exactly one of them. The resource that has run out is not processor or disk but entry points, the number of places a client can arrive.

You could put all five addresses in the name record, since Part II Chapter 4 allows a name to hold several, and clients would pick among them. But consider what happens when one server dies. Its address stays in the record until you notice, and edit the record, and every cached copy of the record around the world expires, and until then clients keep picking the dead machine and waiting for it. Part II Chapter 7 showed you this already: spreading load through names reacts in minutes, when what you need reacts in seconds. You also want the five machines hidden, so that the world sees one thing and you may rearrange whatever stands behind it.

So you put one more machine in front of the five. It has the one address that the name points at, and clients connect to it. For each connection that arrives it chooses one of the five servers and forwards the traffic to it, and forwards the replies back. The world sees one address; you have five servers’ worth of capacity; and the machine in front decides who handles what. What you have built is what the field calls a load balancer. The first question about it is how it chooses, and underneath that lies the question of what it looks at, because two rather different machines are hiding under the one name.

The first kind forwards blind. A connection arrives and the balancer picks a server, perhaps the next one in rotation, perhaps the one with the fewest connections open at the moment. From then on it shovels that connection’s bytes to that server and the server’s bytes back, and it never looks inside them. It does not know whether the bytes are chat or feed, one request or a hundred, HTTP or something else entirely. It works at the level of Part I Chapter 8: it sees four-tuples and forwards them. This is cheap, because there is almost no work per byte, and it is fast, and it can balance anything at all, because it understands nothing. Its limit is that same blindness. It balances whole connections, so if a client opens one connection and sends a thousand requests down it, which is what Part II Chapter 6 told clients to do, keep the connection warm and reuse it, then all thousand requests land on the one server the balancer picked when the connection opened. It cannot spread requests it cannot see.

The second kind understands. This balancer speaks the application protocol, the HTTP of Part I Chapter 18. It accepts the connection itself, reads each request as it arrives, picks a server for that request, forwards the request, reads the response, and sends the response back to the client. Now a client’s thousand requests on one connection can go to a thousand different servers, evenly. And because it reads the requests it can do a great deal more. It can send /feed to the feed servers and /chat to the chat servers. It can retry a failed request on another server. It can count requests per client. The cost is that it does real work per request. It terminates the client’s connection and originates a connection of its own to the server, and that means it also terminates the security from Part I Chapter 21. It holds Courtyard’s certificate and decrypts the traffic, because that is the only way it can read the request to route on it, and it then re-encrypts to the server, or does not. More work, more capability, and it now sees everything.

The blind one is called an L4 load balancer, because it works at the transport layer, which is layer 4 in the model from Part I Chapter 22, and the understanding one is called L7, because the application layer is layer 7. These are the numbers from that model, finally earning their keep. Large systems use both. An L4 layer stands at the very front and spreads raw connections cheaply across a bank of L7 balancers, which then read requests and route them. For Courtyard as it stands, one L7 balancer in front of five servers is the right shape.

The balancer forwards to five servers, and one day one of them crashes. If the balancer keeps sending it a fifth of the traffic, a fifth of Courtyard’s requests go to a dead machine and fail, and the whole reason for having five was to survive one of them dying. The balancer has to find out. So it checks. Every couple of seconds it asks each server a small question, a request to a special path that the server answers only if it is working, and if a server misses a few in a row the balancer stops sending it traffic. When the server begins answering again, the balancer puts it back. This is the heartbeat from Part I Chapter 13 pointed the other way, the balancer pulsing its servers to see which are alive, and it goes by the name health check.

It fails the way Chapter 13 warned that heartbeats fail. A health check answers yes or no, and there is a state it cannot see, which is alive but useless. Consider a server whose disk is failing, or whose event loop is stalled on the one slow thing from Part I Chapter 15, or whose connection to the database has hung. It may still answer the health check, which is a trivial request, while every real request it handles times out. The balancer sees green and keeps feeding it users, and every one of them waits and fails. A naive health check tests whether the process is running rather than whether it is working, and the gap between those two is where the worst outages live. You could make the check exercise the real path, so that it touches the database, but then a brief database hiccup fails every server’s check at the same instant and the balancer concludes that the whole fleet is dead. Health checking is a small art of its own. What it teaches is that up and working are different questions, and a heartbeat only ever answers the first.

Now consider an operation that looks trivial and is not, and that happens every day. You want to remove a server. Perhaps you are deploying a new version, or resizing the machine, or moving it. The server is healthy; you only want it gone. You tell the balancer to stop sending it traffic and you shut it down.

At the instant you shut it down, that server was in the middle of things. It had forty connections open. It was halfway through building a feed for one user, waiting on the database for another, and streaming a photo to a third. All forty of those requests die. Forty people see an error, or a spinner that never resolves, because you turned off a machine that was working, and working on their requests, when you killed it. A healthy server holds live work, and removing it drops that work unless you let it finish first.

So you do not cut it off. You tell the balancer to stop sending the server new connections, but to let the existing ones run. The server finishes its forty requests over the next few seconds, the connections close in the ordinary way, and once the server is holding nothing, then you shut it down. Nobody sees an error, and the removal is invisible. This is called draining the server.

Draining has a time limit, because you will not wait forever for one stuck request, and so there is a grace period after which any remaining work is cut off regardless. It is another timeout, another number with no perfect value. Too short and you cut off requests that were slow but fine; too long and one hung request delays your deploy. And draining interacts badly with long-lived connections, which is the seam this chapter leaves open. A chat connection of the kind from Part I does not finish in a few seconds. It is meant to stay open for hours, and you cannot drain it by waiting, because it will never end on its own. You have to close it yourself and rely on the client to reconnect, through the balancer, to a different server. Draining a fleet that serves requests is easy. Draining a fleet that holds connections is a real problem, and Chapter 10 is where it comes due.

The balancer is now the one address Courtyard has, which makes it the single point of failure that the five servers were supposed to eliminate. If it dies, all five healthy servers are unreachable. So it cannot be one machine. It needs the anycast of Part II Chapter 7, or a pair, or the arrangement of L4 in front of L7, so that the front door is not itself a door that can jam. You have pushed the single point of failure forward rather than removed it, a move you will make several more times.

The balancer also makes its routing decisions on stale information. Its health checks are seconds old, so for a few seconds after a server dies it goes on sending traffic there. Its idea of which server has the fewest connections is a moment behind the truth. It is another small distributed database, this one of which servers are alive and how loaded they are, and like every other one in this part it is slightly wrong all the time.

And there is a decision hidden in the balancer’s per-request spreading that is about to break. If the balancer sends a client’s requests to a different server each time, then no server can remember anything about that client between requests. That is fine, because Chapter 2 made the servers stateless on purpose. But the moment someone puts state back onto a server, per-request spreading delivers the next request to a machine that has never heard of the client. So entry points ran out, and you put a load balancer in front of a fleet of stateless servers, with health checks and draining; the front door itself must now be made redundant, and its view of the servers’ health is always a little stale. The statelessness that all of this rests on was a promise made back in Part I, and in the next chapter it comes due.