Work That Can Wait

Mira uploads a video. Before anyone can watch it, Courtyard has to transcode it into the six resolutions and formats that different devices and connection speeds need, which is the question of Part I Chapter 17, whether a late byte matters, playing out as adaptive video. Transcoding is heavy computation, taking anywhere from thirty seconds to several minutes.

The upload is a request, so the request handler starts transcoding. Thirty seconds later, long past the timeout from Chapter 16 and long past Mira’s patience, the request is still working. It times out, and Mira sees an error, even though the work is proceeding. Worse, the handler has occupied the event loop, or a worker, for thirty seconds, which is the cardinal sin of Part I Chapter 15, and so a handful of long uploads freeze the whole service.

The problem is a mismatch of shape. Request and response, from Part I Chapter 18, is for work that completes in the tens or hundreds of milliseconds a user will wait for. Transcoding is minutes of work. You cannot ask a user to hold a connection open for minutes waiting for the response, since the NAT timeouts of Part II Chapter 3 would drop it anyway, and you cannot tie up a request handler for that long either. The request’s time budget has run out, and some work does not fit inside a request at all.

So you separate accepting the work from doing it. When Mira uploads, the request handler does only the fast part. It stores the raw video, it records that video 91 needs transcoding, and it returns immediately, saying that the upload was received and is being processed. Mira’s request completes in milliseconds. The transcoding itself happens later, on different machines.

The record that work needs doing lives in a durable list of jobs waiting to be done, and that list is a queue. Separate machines, called workers, pull jobs off the queue, do them, transcoding the video, and mark them complete. The request tier and the worker tier are decoupled. Requests produce jobs and workers consume them, and neither waits on the other.

This buys two things beyond merely fitting the work in. The first is durability. The job is in a durable queue, so if a worker crashes halfway through a transcode the job is still there, and another worker picks it up; the work is not lost with the machine. These are the disposable workers of Chapter 2, made safe by a queue that holds the state for them. The second is smoothing. If a thousand people upload at once, a thousand jobs land in the queue, and the workers chew through them at their own sustainable pace. The queue absorbs the spike. It is the buffer of Part I Chapter 14, used deliberately and correctly, and with a bound, because the workers cannot be overwhelmed when they pull work only as fast as they can do it. Backpressure is built in: the queue grows during a spike and drains afterwards, and the workers run flat out but never over. If the queue grows without bound, because uploads persistently outpace transcoding capacity, that is the signal to add workers, and the depth of the queue is the metric that tells you, cleanly, how far behind you are.

Decoupling is not free, and its costs are specific. There are three of them, and then a fourth that nobody sees coming.

The first is that the job runs twice. A worker picks up the job to transcode video 91, does it, and crashes just before marking it complete. The queue, seeing that the job was never confirmed done, gives it to another worker, which transcodes it again. Or a worker is merely slow, the queue concludes that it has died, hands the job to a second worker, and now both are transcoding the same video. This is at-least-once delivery, the honest guarantee of almost every queue. A job will be delivered at least once, and sometimes more, because the only alternative, at most once, would mean dropping jobs whenever a worker’s completion signal is lost, and for a video upload that is worse than doing the work twice. So jobs have to be idempotent, which is Part I Chapter 20 and the whole idempotency thread arriving at its final form. Transcoding video 91 twice has to be safe, and you make it safe by writing the output to a name derived from the video’s id, so that the second run overwrites the first identically instead of creating a duplicate. Every job a queue delivers must be safe to run more than once, because it will be.

You will hear people promise exactly-once delivery, and you should be sceptical. At the level of message delivery over an unreliable network, Part II’s contract, exactly once is impossible; the worker and the queue can always lose the confirmation that passes between them, just as the acknowledgement in Part I Chapter 11 could be lost. What can be achieved is at-least-once delivery plus idempotent processing, which together produce an exactly-once effect. The job may run twice, but the result is as if it ran once. That is the real thing behind the promise, and it is idempotency doing the work, not any magic in the delivery.

The second cost is that the user has to learn that the work finished. Mira’s request returned “processing” at once, and the request is long over, so how does she find out that the video is ready? There are several answers, and all of them are used. Her client can poll, asking every few seconds whether video 91 is done yet. Courtyard can push a notification when it is, over the connection of Part I Chapter 13, held by the connection tier of Chapter 10, so that the finished transcode is delivered like a chat message. Or the whole thing can be fire-and-forget, and the video appears among her uploads when it is ready. Asynchronous work means that the answer comes back on a different channel from the request, and you have to build that channel.

The third cost is that a poison job blocks the queue. One video is corrupt in a way that crashes the transcoder. The queue hands it to a worker, which crashes. The queue redelivers it, since delivery is at least once, and the next worker crashes too. If the queue keeps redelivering the poison job ahead of the others it can jam the whole pipeline, with every worker dying on the same bad job while good jobs pile up behind it. So a job that fails too many times is pulled aside into a holding area for jobs that could not be processed, which is called a dead-letter queue, and the pipeline keeps flowing while a human looks at the poison later. Without this, one bad input is a denial of service against your own workers.

The fourth is the subtle one, and it is the reason this chapter’s flagship figure is a diagram of two operations.

When Mira uploads, the handler does two things. It writes a row to the database saying that video 91 exists, belongs to Mira, and has the status “processing”, and it publishes a job to the queue saying that video 91 needs transcoding. Two operations, two systems: the database, which is Chapter 14’s owned store, and the queue. And the two cannot be made atomic together, because, as Chapter 13 established, no transaction spans two services, and a database and a queue are two systems.

So the database write succeeds, and then the publish to the queue fails, because the queue was briefly unreachable, as Part II’s contract allows. Now the database says that video 91 exists and is processing, and there is no job in the queue to process it. Mira’s video sits in “processing” for ever, because the thing that would have transcoded it was never enqueued. Or the reverse happens: the publish succeeds and the database write fails, and a worker receives a job for a video that the database has no record of. The two systems disagree, permanently, because the process died in the gap between two writes that had to both happen or neither.

The fix is one to know well, because the pattern recurs. You do not publish to the queue directly. Instead, within the same database transaction that writes the row for video 91, you write a second row to a table called the outbox, saying that a message to transcode video 91 needs publishing. Because it is the same transaction, the database guarantees that the video record and the outbox row commit together or not at all; this is the atomicity you do have, within one database. Then a separate process, the relay, reads the outbox table and publishes each entry to the queue, marking it done once the queue confirms. If the relay crashes, it re-reads the unpublished outbox rows and tries again, which is at-least-once delivery once more, so the job has to be idempotent, and the thread closes. A problem of atomicity across two systems has been converted into a problem of atomicity within one system, which the database solves, plus a relay that delivers at least once, which idempotency covers. The field calls this the outbox pattern, and it is the standard answer to writing to a database and publishing an event atomically, which you will need constantly from the moment you have both a database and a queue.

An asynchronous system is harder to reason about than a synchronous one. The work happens somewhere else, later, and whether it worked is no longer answered by the response. Jobs can be delayed, when the queue is backed up; duplicated, since delivery is at least once; stuck, when a poison job lands in the dead-letter queue; or lost, if idempotency or the outbox is missing. What the user sees is eventual completeness. The video will be ready, but when is not guaranteed, only that it will happen eventually, and that is a promise you have to design the product around, by showing “processing” rather than pretending the work was instant. And the queue itself is now critical infrastructure, because if it is down no asynchronous work happens at all, so it inherits Chapters 3 and 5, replication and durability, like every other stateful thing in this part. The seventeenth revision of Courtyard moves long work to a durable queue processed by workers, and accepts that jobs arrive at least once and so must be idempotent, that completion arrives on a separate channel, that poison jobs need a dead-letter queue, and that writing and publishing atomically needs the outbox pattern.