Everyone Is Online at Once

Everything since Chapter 6 has been about the feed, which is to say about requests that arrive, get answered, and end. The stateless fleet, the caches, the edge, all of it assumed the request-and-response shape from Part I Chapter 18, in which a connection carries one exchange and may then close. Chat is not that shape, and at scale the difference becomes a different architecture.

Recall Part I Chapter 13. A chat client holds a connection open, for hours, so that Mira’s message can be pushed to you the instant she sends it. Now multiply by Courtyard’s users. A million people have the app open, which is a million connections, held open, at this moment, and almost all of them silent, because at any instant almost everyone is reading rather than typing. A million connections, and perhaps ten thousand of them carrying a byte in any given second.

The feed’s servers cannot do this. A feed server handles a request in a few milliseconds and moves on; it touches thousands of connections an hour but holds almost none. A chat server holds a million connections that do almost nothing but must stay open. That is not more of the same load. It is a different resource altogether. What has run out is not processor or disk but the ability to hold connections open, and it needs its own tier of machines, built differently.

Go back to the fork of Part I Chapter 9, threads or the event loop. A thread per connection means a million threads. Each thread needs memory for its stack, a megabyte or so, so a million threads is a terabyte of memory doing nothing, since the threads are almost all parked. The operating system has to schedule among a million threads. It falls over long before a million; it struggles at ten thousand. This is the wall that Part I Chapter 9 named, the reason the event loop exists, and this is where it carries the weight.

The event loop holds a million connections as a million entries in a list, watched by one thread, or one per core, and the idle ones cost almost nothing. The nine hundred and ninety thousand silent connections are only entries; the ten thousand active ones are what the loop actually processes. One machine can hold hundreds of thousands of connections this way, and a modest fleet holds a million. The loop was introduced in Part I as the shape of a server. Here it turns out to be the only thing that makes chat at this scale physically possible.

Chapter 7 told you to move all state off the servers so that any server could handle any request. A connection breaks that rule, and the break cannot be mended, because a connection is state that physically lives on one machine. The bytes from your phone are flowing, through the whole tower of Part I, to one particular chat server, the one that terminated your TCP connection. You cannot put a live TCP connection in a shared store, and you cannot hand it to another machine. The four-tuple, the sequence numbers, the send buffer, from Part I Chapters 8, 11, and 14, are resident on that one server by physical necessity.

So the connection tier is the exception that Chapter 7 promised. Every user is pinned to whichever chat server they happen to be connected to, not by a table of sticky sessions but by the plain fact that their bytes go there. And this brings back the problem from Chapter 6 with teeth. To take a chat server out of service you cannot wait for its connections to end, because they do not end; they are meant to last for hours. You have to close them yourself and rely on the clients to reconnect, through the load balancer, to a different chat server. A deploy of the chat tier is therefore a rolling wave of forced reconnections, and if you do it carelessly, closing a hundred thousand connections at once, then a hundred thousand clients reconnect at once, and you have stampeded your own connection tier. So you drain slowly, a slice at a time, with jitter, which is Chapter 8’s lesson again. Every chat platform’s deploy is shaped by this.

Now for the problem that makes chat hard, and it appears only at this scale. You send Mira a message. Your connection is on chat server 7. Mira’s connection is on chat server 34. Server 7 has your message and needs to push it down Mira’s connection, which it does not have. Mira is not on server 7, and server 7 does not even know where she is.

On one machine, in Part I, this was trivial. Mira’s connection was in the same process, in the same list of handles, and the server wrote to it. Now there are five hundred chat servers, Mira is on exactly one of them, and the server holding your message is almost certainly a different one. For the first time in this book, Courtyard’s servers have to talk to each other. Not to the database; to one another, because the two ends of a conversation are on two different machines and something has to bridge them. Two things are needed.

The first is a record of who is connected where. When Mira’s phone connects to server 34, server 34 writes it down: Mira is on server 34. The record goes in a shared, fast store, the store of Chapter 7 and the in-memory tier of Chapter 8, that every chat server can read. When Mira disconnects, server 34 removes the entry. This is a presence registry, and it is also, not by coincidence, the same information as the online dot from Part I Chapter 13: who is connected, and where. It is a distributed database of the location of live connections, and it has all the usual troubles. It is slightly stale, since Mira may have just disconnected and the entry not yet been removed. It must be replicated, as in Chapter 3. And a wrong entry sends a message to a server that no longer holds Mira.

The second is a way for one server to hand a message to another. Server 7, holding your message, looks Mira up in the registry and finds server 34. Server 7 sends the message to server 34, a server-to-server call over the network, Part I’s tower between two of Courtyard’s own machines, and server 34, which does hold Mira’s connection, pushes it down. The message hopped: your phone to server 7, server 7 to server 34, server 34 to Mira’s phone. Two of Courtyard’s servers cooperated to bridge two connections of which each held one end.

A group chat is the same thing fanned out. Look up all hundred members, group them by which server holds each, and send each server the message once for all the members it holds. A message to a large group becomes a handful of server-to-server sends, each delivering to many local connections. And a message to a member who is offline, with no entry in the registry, is not delivered live at all. It is stored in the database for them to fetch when they reconnect, which is the other half of why chat needs both a connection tier and durable storage.

Part I Chapter 15 gave the event loop two rules, never block it and never outrun it, and warned that these were the failures you would ship. At a million connections they are not bugs but catastrophes. Take the first rule. One chat server holds two hundred thousand connections on one loop. A single handler does one slow thing, a database call without an await, a heavy computation, anything from the list in Part I Chapter 15, and the loop stops, and two hundred thousand people are frozen until it resumes. Not one user; two hundred thousand, because all of them are served by the one loop that is now stuck. The blast radius of a blocked loop is the whole machine’s connection count. This is why chat servers are ruthless about doing no real work in the connection handler. They receive a message and hand it at once to something else, the registry, another server, the database, and they never compute in the loop, because a hiccup is a mass freeze.

Take the second rule. A slow client, Theo from Part I Chapter 15 on bad Wi-Fi, or the phone from Part II’s sidebar, cannot receive messages as fast as a busy group produces them. The server must not buffer without bound, since that is the balloon of memory that kills the server for everyone, and it must not block the loop waiting for him. So it bounds Theo’s outgoing queue, and when the queue fills it drops or disconnects him, which is Part I Chapter 14’s three choices made per connection, a million times over. Backpressure, from Part I, is now what stands between one slow phone and a server-wide collapse of memory.

The registry is a new critical dependency. If it is wrong or down, messages cannot find their recipients. It is stale by nature: someone reconnects to a new server faster than the registry updates, a message goes to their old server, which no longer has them, and the message has to be re-routed, or dropped and fetched again. Server-to-server delivery adds a hop that can fail on its own. Server 7 finds Mira on server 34 and sends the message, and server 34 dies before delivering it. Did Mira get it? This is the idempotency and acknowledgement of Part I Chapter 20, now between two of your own servers, and the message needs an acknowledgement from the client, Mira’s phone confirming receipt, or it has not been delivered, only sent.

The connection tier is also a scaling axis independent of everything else. Feed traffic scales with requests. The connection tier scales with simultaneous users, which is a different number that grows differently; a celebrity going live can add a million connections in minutes without adding a single feed request. You now size two fleets against two unrelated demands.

So connections ran out, and you built a dedicated connection tier on the event loop, with a presence registry and server-to-server delivery of messages. The registry is a stale critical dependency, delivery adds a hop that needs its own acknowledgement, a blocked loop now freezes hundreds of thousands of people at once, and connections are pinned state that turn every deploy into a careful wave of reconnections. Messages find their recipients. The feed, which has been waiting since Chapter 6, has a harder question of its own.