The Answer Takes a Minute

Courtyard adds a feature: a model you can talk to. You type a question and it writes back an answer, in full sentences, thoughtfully. Everything about the shape of that request is unlike anything else in the book, and to see how, it helps to line it up against the two payloads that ran through all three parts.

Chat is tiny, bidirectional, bound by latency, impossible to cache, and carried over a long-lived connection. A message is a hundred bytes. What matters is the round trip, and the connection stays open for hours, as in Part I Chapter 13 and Part III Chapter 10. Video is enormous, one-directional, bound by throughput and jitter, cacheable, and pushed out to the edge. A file is millions of bytes. What matters is bandwidth and smoothness, and a copy near the viewer is as good as the original, as in Part I Chapter 17 and Part III Chapters 8 and 9.

A model’s reply is neither of these. The request is tiny, a sentence. The response is medium-sized, a few kilobytes of text. But between them sits thirty seconds of the heaviest computation in the entire system, on hardware that costs more per hour than a rack of ordinary servers, and the response does not arrive as a single blob at the end. It arrives a piece at a time, as it is computed, a word every few tens of milliseconds, streamed over a connection that has to stay open for the whole thirty seconds. It is slow in the way that video is long, but the slowness is computation rather than size. It is interactive in the way that chat is, but a single exchange lasts a minute. It is a third shape, and it stresses the tower in a third place.

Because the response streams over a connection held open for thirty seconds, three rungs from Part I that were handled without much fuss now become the whole game.

The first is keepalive, without which the reply dies in the middle. A connection that sends a token every fifty milliseconds is active. But between two tokens, if the model pauses to think, the connection can be silent for a second or two, and every intermediary you built in Part II and Part III is watching for that silence. The NAT box of Part II Chapter 3 forgets idle connections. The load balancer of Part III Chapter 6 has an idle-connection timeout and will cut a connection it believes to be dead. The gateway of Part III Chapter 15 has one too. A thirty-second streamed reply passes through all of them, and any one of them, seeing a two-second gap, can decide the connection is dead and kill it, and the user’s answer stops halfway through a sentence. So the stream must be kept visibly alive. The tokens themselves are the heartbeat while they flow, and when the model pauses, the server sends the keepalive pings of Part I Chapter 13, so that no box in the path mistakes thinking for death. Every timeout on every intermediary, all the way down, has to be set in the knowledge that this payload is supposed to be slow and silent in bursts, which is the opposite of what those timeouts were tuned for.

The second is backpressure, without which the server wastes its scarcest resource. The model generates tokens at some rate, and the user’s connection carries them away at some rate. If the user is on a slow phone, as in the sidebar to Part II, the connection cannot drain tokens as fast as the model makes them, and by Part I Chapters 14 and 15 the tokens back up in the server’s buffer, which grows. What makes this acute is that the thing generating the tokens is the most expensive hardware in the building. If it is blocked on a slow client’s backpressure, it sits idle, generating for no one, while it waits for one phone to catch up. You cannot let a slow reader stall an accelerator. So the server separates generation from delivery. The model generates into a bounded buffer at full speed, freeing the accelerator the instant it is done, and a cheap ordinary server drains that buffer to the slow client at the client’s pace. This is the backpressure of Part I with a twist the book has not shown before: the backpressure must not reach all the way back to the scarce resource, because the scarce resource is too precious to wait. You absorb it in a cheap tier so that the expensive tier never feels it.

The third is a timeout that tells slow from dead, on a request that is supposed to be slow. The whole apparatus of Part III Chapter 16 assumed a request that should be fast and a timeout that catches it being slow. Here the request should take thirty seconds, and a timeout tuned for a normal request would kill every single one. So the timeout has to distinguish a model that is working, with tokens flowing, from a model whose machine died in the middle of generation, and it cannot do so by total duration, because the total duration is supposed to be enormous. It does so by progress. As long as tokens keep arriving, the request is healthy however long it runs. If tokens stop arriving for longer than the model’s longest plausible pause, then it is dead. The timeout watches the gaps between tokens, not the total. The deepest lesson of Part I Chapter 13, that you cannot tell slow from dead by silence alone, meets its purest case here, because this is the one payload where long silence is normal and you must still catch real death.

Now to the part that is new, and it inverts the whole scaling logic of Part III. Every resource in Part III could be added to. Out of processor? Buy machines, as in Chapter 1. Out of read throughput? Add replicas, as in Chapter 3. Out of entry points? Add servers behind a balancer, as in Chapter 6. The whole part was a series of running out and adding more, and it worked because the things that ran out were replicable, ordinary machines you can rent by the thousand.

The accelerators that run the model are not like that. They are expensive. They are supply-constrained, so that you cannot always buy more at any price, because the world’s supply is finite and spoken for. And even when you can buy one, it costs more per hour than a whole rack of the servers Part III scaled with. So the defining constraint turns over. In Part III, load was the problem and hardware was the cheap, elastic answer. Here, hardware is the fixed, scarce thing, and load must be shaped to fit it. You cannot meet demand by adding accelerators the way you added web servers. There are only so many, and they are always, by design, nearly fully used, because idle time on the most expensive hardware you own is the most expensive thing there is.

This means that the one thing you cannot do is what Part III did by default, which was to accept all the load and scale to meet it. You have a fixed generation capacity, demand that exceeds it, and no elastic answer. So you must do the thing Part III mostly avoided. You must turn work away, deliberately and well.

You do it by putting a bounded queue in front of the accelerators. A request to the model does not go straight to an accelerator. It goes into the queue, and accelerators pull from the queue as they finish their previous work, running at full utilisation always, since an idle accelerator is the cardinal waste. This is the queue of Part III Chapter 17, which smoothed a spike of asynchronous work, but with a sharper purpose. It is not only smoothing. It is rationing access to a resource that cannot be scaled up.

And because the resource is fixed and the demand is not, the queue fills. When it does, you are back at the three choices of Part I Chapter 14 and the load shedding of Part III Chapter 16, and you must choose. You can admit the request, if there is capacity to run it. You can queue it, if there is no capacity now, in which case you tell the user so, because a minute of silence followed by a minute of queue is two minutes and they will leave. Or you can shed it, when the queue is at its bound, rejecting it cleanly with a suggestion to try again shortly rather than accepting work you cannot do and failing it slowly. The field calls this admission control: deciding at the door who gets in, because the room is full and cannot be made bigger. Part III shed load only under failure. Here you shed it under normal operation, continuously, because demand structurally exceeds a supply you cannot grow.

Admission control also forces a question the rest of the book did not have to answer, which is who gets in. When you cannot serve everyone, fairness becomes a design decision. First come, first served is simple, and it lets one heavy user monopolise the accelerators. Per-user limits keep any one user from starving the rest, but they waste capacity when the accelerators would otherwise be idle. Priority tiers, with paying users first or interactive requests ahead of batch ones, express a business choice in the queue’s ordering. There is no neutral answer. A fixed resource under excess demand must encode a policy about who waits, and pretending otherwise only means the policy is accidental. These are the dials of Part III, with no right answer and only a trade to tune, arriving at the one resource where the trade cannot be avoided.

Step back for a moment, because this is why the chapter is a bonus and not a fourth part. Nothing here is a new rung. Keepalive is Part I Chapter 13. Backpressure is Part I Chapters 14 and 15. A timeout that tells slow from dead is the deepest lesson of Part I Chapter 13. The queue is Part III Chapter 17. Admission control is the load shedding of Part III Chapter 16, made routine. Every mechanism the model-serving system needs, you already built, for chat and for video and for the general case of many machines. The tower did not change.

What changed is which rungs dominate. Chat lit up the connection tier and the round trip. Video lit up bandwidth and the edge. The model’s reply lights up the rungs that were most subtle, the ones about long silences that are not death, about slow readers that must not stall fast producers, about a resource you cannot buy more of, and it lights them because a single payload finally combines long duration, streaming delivery, and irreplaceable scarcity in one request. Give the same tower a third kind of load and a third set of rungs turns out to matter most.

That is the whole lesson, and it seems to me the right note to end on. You did not learn fifty mechanisms for fifty situations. You learned one tower, built from necessity, and the situations, chat, video, a model that thinks for a minute, are lenses that each bring a different part of it into focus. The next payload, whatever it is, will be a fourth lens. It will not need a new tower. It will need you to look at the one you have and see which rungs it lights up.