One Slow Thing

The camera turns around again, away from the bytes on the wire and towards the program at Mira’s end that receives them. Courtyard’s server is holding thirty conversations in the loop from Chapter 9, and it is doing well. Over the course of this chapter it does three ordinary things, and each of them breaks every conversation at once.

You send Mira a photo in the group chat. Her server receives it, reliably and in order thanks to Chapter 11, and now has something to do with it. It must decode the compressed image, make a small thumbnail to show in everyone’s chat window, and send the thumbnail on. Decoding a three-megabyte image and shrinking it is real work. Let us call it two hundred milliseconds of the processor’s full attention.

Her program does the natural thing. When the socket carrying your photo becomes readable, the loop reads it, and then, right there inside the handler, decodes it, shrinks it, and moves on.

forever:
    ready = wait until any of handles has something
    for each h in ready:
        bytes = read from h
        if bytes are a photo:
            image = decode(bytes)          ← 200 ms
            thumb = shrink(image)          ← inside the loop
            send thumb to everyone
        else:
            ...

For two hundred milliseconds the program is decoding. It is not at the top of the loop, and it is not asking what is ready. Theo’s message arrives and sits in a buffer. Priya’s heartbeat from Chapter 13 arrives and sits. A new connection knocks, and it sits too. Twenty-nine conversations are frozen for a fifth of a second because one of them sent a picture.

Two hundred milliseconds does not sound like much. But suppose it is a busy evening and photos arrive every second. The loop now spends a fifth of every second decoding, and during each of those fifths nothing else happens, so everyone’s messages arrive in lurches. Then someone sends a large photo, one that takes a full second to decode, and for that second the server is, to everyone connected to it, gone. Heartbeats go unanswered. Some clients decide the server is dead and reconnect, which is more work for the loop, which makes the next lurch longer than the last.

loading…

Mira fixes the photo problem, in a way we will come to shortly, and then adds a feature. When you send a message, the server looks up your display name and avatar in a small database on the same machine, so that it can attach them before forwarding the message on.

The database is fast. A lookup takes perhaps five milliseconds. Her handler calls it and waits for the answer.

        record = database.lookup(sender)    ← 5 ms, inside the loop
        forward message with record.name

Five milliseconds is a trivial amount of time. But a lookup is a read, whether from a file or from a socket connected to a database process, and a read, as Chapter 9 explained at some length, parks the program until there is something to read. For those five milliseconds the loop is not at the top. With thirty conversations and five milliseconds spent on each message, the loop can process at most two hundred messages a second, however fast the machine underneath it is. Then one day the database is slow, because the disk is busy or because it is doing some maintenance of its own, and the five milliseconds becomes five hundred, and the server is gone again for everyone, because one lookup for one person is taking a while.

I think this is the failure that ships most often, and it is the hardest to see, because nothing in the code looks slow. Nobody writes a giant image decode by accident. But everybody writes a lookup, and a lookup is a wait, and a wait inside the loop is the loop not looping.

Mira fixes that too, and then runs straight into Chapter 14.

Theo is on a terrible connection tonight. His machine reads slowly, his window is small and often zero, and writes to his socket block. Mira’s server, forwarding a busy group chat to thirty people, reaches Theo’s socket, calls write, and parks. The loop is stuck on Theo, and twenty-nine people wait for Theo’s window to open.

So Mira does what Chapter 14 warned against doing without thought. She makes the socket non-blocking, so that write returns immediately with however much fitted, and she keeps whatever did not fit in a queue inside her program, one queue per client, to be sent later. Now writes never park. The loop keeps looping, and everyone else is fine.

And Theo’s queue grows. Every message in the group chat goes into it, because his window is closed more often than it is open. A kilobyte a second goes in and a hundred bytes a second come out. After an hour his queue is several megabytes. After a night it is hundreds. Mira’s server has a slowly inflating balloon of memory attached to one bad connection, and one morning the balloon is large enough that the machine runs out of memory and the whole server dies, for everyone, because Theo’s Wi-Fi was bad.

loading…

She did not solve the problem from Chapter 14. She chose the third of its options, which was to buffer the bytes herself, and it did what that chapter said it would do.

So we have three ordinary things, a decode, a lookup and a slow client, and three different fixes that are all wrong in the same way. Go back to the agreement from Chapter 9. The operating system’s half was that it would tell you when any handle was ready. Your half was that you would never park yourself on one thing. The decode parked the loop on a computation. The lookup parked it on a read. The slow client either parked it on a write or, worse, did not park it at all and instead let a queue grow without limit. In every case the loop stopped being a loop. It was meant to be a cursor that touches each ready thing briefly and returns to the top, and instead it lingered.

The loop is for waiting. It is not for working. The moment you do something inside it that takes time, whether that is computing, or reading, or writing to something that is full, you have turned a program that watches thirty conversations into a program that watches one, and the other twenty-nine are frozen for the duration. That is the rule, and it has two halves, matching the two ways of lingering.

The first half says that anything which takes time goes somewhere else. For the decode, you hand the bytes to a worker, which may be a separate thread, or a separate process, or a pool of either, and you go straight back to the top of the loop. The worker decodes at its own pace, on its own processor core if there is one to spare, without the loop waiting for it. When it finishes, it needs to get the thumbnail back to the loop, and it does this by becoming an event. The worker’s completion is one more thing the loop can wait on, alongside the sockets. The loop wakes, sees that a thumbnail is ready, sends it, and goes back to waiting. Two hundred milliseconds of work happened, and the loop was parked at the top for all of it, attending to everything else.

For the lookup, the treatment is the same in a different guise. A database lookup is a read on a socket, and reads on sockets are what the loop is for. You do not call the lookup and wait for the answer. You send the query, register the database’s socket as one more handle, and go back to the top. When the answer arrives the socket becomes readable, the loop wakes, and the handler continues from where it left off. The five milliseconds, or the five hundred, pass with the loop attending to everyone else.

This is what every await in every modern language is doing. To await the lookup means to send the query, park this handler rather than the loop, and let the loop run other handlers until the answer arrives. The handler reads like a straight line. Underneath, it has been cut into pieces at each await, and the loop is running the pieces between other people’s pieces. That is also why it is so easy to break. A slow call without an await in front of it does not park the handler. It parks the loop.

The second half of the rule says that writes get the same treatment as reads. A socket whose window is closed is not writable, in the same sense that an empty socket is not readable. The operating system will tell you when it becomes writable; this is the same readiness call from Chapter 9, asked about the other direction. So the loop should not call write and hope. It should write what fits, and if not everything fitted, stop writing to that socket and ask the loop to wake it when the socket is writable again. Then it sends the rest.

But the bytes that did not fit have to live somewhere until then. That somewhere is a queue, per client, inside the program, which is the very thing that killed Mira’s server. The difference is a single word: the queue is bounded. Give it a limit. When Theo’s queue reaches the limit, you are facing Chapter 14’s three choices again, and this time you face them honestly. You can wait, and stop reading from the group chat until Theo catches up, which slows everyone to Theo’s pace. You can drop, and throw away some of what Theo would have received, which is fine for a presence update and not fine for a message. Or you can disconnect Theo, on the grounds that a client who cannot keep up is a client who has effectively gone.

Any of the three is defensible. What is not defensible is the fourth thing Mira did, which was to not decide, and to let the queue grow until the machine’s memory made the decision for her at four in the morning.

The backpressure from Chapter 14 was the operating system pushing slowness back up to your program. The second half of the rule is your program pushing it further back, to its source. If a client cannot keep up, something upstream has to know. A system in which every queue is bounded and every producer respects its consumer’s limit degrades gracefully, with the slow parts getting slow and the rest carrying on. A system with one unbounded queue anywhere in it falls over, eventually, all at once.

You are going to write servers. Most of them will be built on a loop you did not write, in a language that hides it behind async and await, and it will be very easy to forget that the loop exists. So the ways in which you will break it deserve to be named in advance.

The first is the slow call in the handler: a database query, a file read, a request to another service, made synchronously, with no await, inside a handler. The loop stops for the duration, and every user of your server freezes. It works fine in testing, because the database is fast and there is one user.

The second is the computation in the handler. Image resizing, compression, parsing a large document, cryptography, anything that uses the processor for more than a millisecond or two. It does not matter that this is not input or output. The loop is not looping while it runs.

The third is the unbounded write queue. One slow client, one queue with no limit, and one server that dies for everyone after an hour or a night. It is usually discovered when someone looks at the memory graph and asks why it only ever goes up.

Under all three sits a belief, which is that async means fast. It does not. Async means that the loop keeps looping while you wait. A slow lookup, awaited properly, is still slow, and Mira’s user still waits five hundred milliseconds for their name to be attached. What they no longer do is make everyone else wait. Async is about isolation rather than speed. It lets one slow thing be one slow thing instead of thirty.

Courtyard’s server keeps its promise now. Photos are decoded by workers. Lookups are awaited. Slow clients get a bounded queue and, when it fills, a decision. The loop touches each ready thing and returns to the top, and thirty conversations, or thirty thousand, proceed at their own pace without freezing one another.

Two chapters ago the window taught you that the receiver can be slower than the sender. There is one more party that can be slower than the sender, and it is not the receiver at all.

Coming up · I·16 — The Network Is Not Yours

Mira’s window is wide open and pieces are still going missing. The thing that is full is not her machine. It is a box in the middle, and it belongs to someone else.

→