The Server Remembers

A developer ships a pleasant feature. Courtyard should remember where you were: which conversation you had open, that you are logged in, that you have dismissed the notification banner. The straightforward way to build it is for the server, when you log in, to make a small record of your session in its own memory, hand you an identifier for that record, and look the record up on each request you make afterwards. It works perfectly in testing, on one server.

In production, behind the load balancer of Chapter 6, users begin to be logged out at random. Not all of them, and not predictably. Every so often a request goes out and the login screen comes back. What has run out this time is not a machine or a disk. It is the statelessness you were promised in Part I, and to see how it ran out you have to follow one user’s requests.

The load balancer spreads requests across five servers, one request at a time, evenly, which is the understanding balancer of Chapter 6 doing what it was built to do. You log in, and that request happens to arrive at server 2, which makes your session record in its own memory. Your next request arrives at server 4. Server 4 looks in its memory for your session and finds nothing there, because it has never heard of you; your record is on server 2. So server 4 concludes that you are not logged in and sends you to the login screen. The session lives on one server and the requests are spread across all five, so four times out of five the server handling your request is not the server holding your session. Nothing has failed. The machine that remembers you is not the machine you reached.

This is the promise of Part I Chapter 20 coming due, and coming due as a punishment for having broken it. Back then you chose to make the server stateless, to have it remember nothing between requests, and it seemed a small and almost arbitrary discipline. Chapter 6 relied on it without naming it. Per-request spreading works only if the servers are interchangeable, and they are interchangeable only if none of them remembers anything the others do not. The moment this developer put a session into one server’s memory the servers stopped being interchangeable, and everything that stood on them, the load balancer, the fleet, the ability to add and remove machines, broke without making a sound.

The obvious repair is to make the balancer send each user back to the same server every time. You logged in on server 2, so all your requests go to server 2, which has your session. The balancer keeps a table of which user belongs to which server and honours it. This arrangement is called sticky sessions, and it works, and it throws away everything that Chapter 6 built.

Server 2 can no longer be removed without logging out everyone stuck to it. The drain from Chapter 6, which lets work in flight finish and then closes cleanly, does not help, because the sessions are not work in flight. They are resident, and they exist only in server 2’s memory. Deploy a new version, which means restarting every server, and every user on every server is logged out at once. A deploy, the thing you do several times a week, has become a mass logout.

Load also stops being even. A server that has been up longer has accumulated more stuck users, while a server that has just restarted has none. The popular users, the busy ones, are stuck wherever they happened to land, so one server ends up carrying all the heavy sessions while another idles. You have brought back the hot-shard problem of Chapter 4, in the application tier this time, for no reason except that state was put in the wrong place.

Sticky sessions treat the symptom, and the symptom is that the state is on the server. If the state is taken off the server, the symptom goes with it. The session has to live somewhere that every server can reach, so that it no longer matters which server handles a request, and there are two such places. They are different in ways that matter.

The first is a shared store. Put the sessions in a separate, fast data store that all five servers can read and write. When you log in, whichever server handles it writes your session there, and any server handling any later request reads it from there. The servers are interchangeable again, because none of them remembers you; the store does. Chapter 6’s fleet works as designed. Deploys log out nobody, because the sessions are not in the servers. Load is even, because any server can serve anyone. The store has to be fast, since it is read on every request, so it is usually an in-memory store, the subject of Chapter 8, rather than the main database. And it is now a thing every request depends on, so it has to be replicated and made reliable itself, or it becomes the new single point of failure. If the session store is down, everyone is logged out at once. You have moved the state to one shared place, which is cleaner, and you have also made that place critical.

The second is to store no session on the server at all and put it in the client instead. When you log in, the server hands you a token that contains the facts, who you are and when this expires, together with a cryptographic signature of those facts, keyed with a secret only the servers know. This is the integrity from Part I Chapter 21. Your client sends the token with every request. Any server checks the signature, and if the signature is valid the server knows the token’s contents are genuine, because only a server could have produced that signature. It reads who you are straight from the token and proceeds. There is no lookup and no shared store. The state is in the client’s pocket, and the signature is what makes it safe to trust something the client is holding. This is a signed token.

Both of these take the state off the server. They differ on one axis, which is who holds the state and therefore who can take it back. The shared store holds the truth, so the server can revoke a session at once: delete the record, and the next request fails and the user is logged out. That is essential when a stolen session has to be killed now, or an account banned with immediate effect. The cost is a lookup on every request and a store that must always be up.

The token puts the truth in the client, so there is nothing to look up and nothing that must be up; the token verifies itself. But it cannot easily be un-issued. A token signed to be valid for an hour is valid for an hour, on every server, even if you want it dead this minute, because the servers check nothing except the signature and do not know which tokens you have decided to revoke. To revoke a token you would need a list of revoked tokens that the servers consult on each request, and that list is a shared store, brought back for the one case the token was meant to avoid. So tokens are kept short-lived, to bound the damage, and paired with a longer-lived credential that is checked against a store whenever a new token is issued. Most real systems land on that hybrid. As a rule, tokens serve when you can tolerate a short delay in revocation and want scale and simplicity, and a shared store serves when you need control at once. Courtyard probably uses tokens for ordinary sessions and a store for the security-critical parts, which is the common answer.

Sessions were the example. The rule is larger, and it is the organising principle of the whole application tier. Anything a server remembers between requests is a thing that breaks scaling, deploys, and failover, so servers remember nothing, and every piece of remembered state lives in a store built to be shared. An uploaded file half received and held in one server’s memory breaks when the next chunk arrives at another server. A running total accumulated across requests is wrong the moment requests are spread. A cache of expensive results in one server’s memory is fine as an optimisation and never as a source of truth, because four-fifths of requests will not find it. Each of these is the session bug in a different hat. The discipline is to ask, of anything a server holds, whether the system would still be correct if the next request went to a different machine, and if it would not, to move the thing out.

The one thing that cannot be moved out is a live connection. A chat socket of the kind from Part I is state, and it is resident on one server by physical necessity, because the bytes are flowing to that machine. You cannot put a TCP connection in a shared store. That is the exception the rule cannot absorb, it is why chat scales differently from everything else, and it is Chapter 10.

If you chose the shared store, it now stands on the critical path of every request, and its failure is a total outage. It therefore inherits Chapters 3 and 5, replication and the partition choice, and you are running another distributed database whether you wanted one or not. If you chose tokens, you have accepted that revocation is delayed by the token’s lifetime, and you have taken on the job of protecting the signing secret, because anyone who steals it can forge a token for any user and the whole trust argument of Part I Chapter 21 collapses. A leaked signing key is a total compromise, and a silent one, until you rotate it.

So the statelessness promise came due, and you moved session state out of the servers into a shared store, or into signed tokens, or both. Either the store is now critical and must be made reliable, or the signing key is now critical and must be protected. And live connections remain the one kind of state that cannot be sent anywhere else, which is a debt this part has not yet paid.