Everything Is Always Broken

Courtyard is now a dozen services, spread across two regions, running on some three thousand machines, and the arithmetic of failure at that size is worth doing. Suppose a machine fails once a year, which is a kind estimate. Three thousand machines then produce a failure roughly every three hours, and that is before you have counted disks, network links, deploys, and dependency blips, all of which raise the rate. At this scale something is always broken. There is no moment at which every machine and every link and every service is healthy at the same time. When the system was one machine a failure was an event; now it is the weather. The failure budget has run out, since the system can no longer treat failure as an exception, and it has to work anyway.

Working anyway means that every service, on every one of the network calls of Chapter 13, has to assume the call may fail and know what it will do when it does. This chapter is that plan, and much of it concerns how the plan everyone reaches for first makes matters a great deal worse.

Before you can react to a failure you have to notice one, and that is the question of Part I Chapter 11, which never had a good answer. The feed service calls the accounts service, and accounts does not answer. It may be down, or it may only be slow, and from where the feed service sits a dead service and a slow one look the same, which was the lesson of Part I Chapter 13 and is the truth that recurs most often in this book. So the feed service waits, and at some point it decides that the call has failed. That decision, made after a chosen interval, is the timeout.

Choose the interval too short and you give up on calls that were about to succeed, converting slow into failed by your own hand. Choose it too long and a hung dependency holds the feed service’s own resources for the whole interval. The event-loop handler from Part I Chapter 15 sits waiting, and if accounts is slow for everybody then every feed request is sitting the same way, and the feed service runs out of capacity waiting on a service that is never going to answer. There is no right value, only a balance, and it is the balance of Part I Chapter 11 again, except that now it has to be struck on every one of thousands of calls between services.

There is a twist that only appears at this scale. A user’s request enters at the gateway and fans out. The gateway calls feed, feed calls accounts, accounts calls the cache, and so on ten services deep. If each service has its own independent timeout, the ones deep in the graph keep working on a request the user abandoned three seconds ago, because nothing told them. So the deadline is set once, at the edge, where the gateway decides that this request may take three seconds in total, and then it is passed down with every call. The gateway tells feed that it has three seconds. Feed, having spent one, tells accounts that it has two. Accounts tells the cache that it has one. Any service that finds its share already spent stops at once, because the user is gone and whatever it might produce is worth nothing. Nobody works on a dead request. The field calls this deadline propagation, and a service graph that lacks it spends an enormous share of its effort computing answers that no one will ever read.

A call has failed, then, and the response that everyone reaches for is to try again. The failure may well be transient, since Part II’s contract says that pieces get lost and blips happen, and a second attempt may succeed. So the feed service retries its call to accounts.

For a single failed call this is the right thing to do, and it is the territory of Part I Chapter 20, with the condition that chapter attached: only if the call is idempotent. Retrying getUser is safe, because it is a read. Retrying postMessage may post the message twice, unless it carries the idempotency key from Chapter 20. Retries therefore require idempotency, which is why that thread runs through the whole book. At scale everything gets retried, so everything that gets retried has to be safe to retry, which means an idempotency key on every operation that is not idempotent by nature. That is the precondition. With it in place, let’s watch the fix become the disaster.

Accounts has a brief slowdown, a garbage-collection pause or a slow query or a momentary spike in traffic, and is sluggish for two seconds. Nothing about this is serious, and left alone it would recover.

But accounts is called by feed, by chat, by search, by notifications, by a dozen services in all, each of them serving thousands of requests, and all of those calls are now slow. All of them reach their timeouts, and all of them retry. So in the two seconds during which accounts is struggling it receives not its normal load but its normal load plus a retry of everything that timed out, which is twice or three times the requests. Accounts, already slow, is now under triple traffic, which makes it slower, which makes more calls time out, which produces more retries, which makes it slower still. The retries are feeding the fire. A two-second blip that would have healed itself has become a service pinned under three times its capacity, and it cannot recover, because the moment it serves one request two more retries arrive to take its place. The retry, which was the fix for failure, has converted a transient slowdown into a sustained outage. And since a dozen services depend on accounts, and further services depend on those, the outage spreads outward through the call graph. This is a cascading failure, the whole system down because one service hiccuped and everybody helpfully tried again.

What you have just watched is a retry storm, the most important failure in this part, because the cause is the cure. The instinct that is correct for one call is catastrophic in aggregate. Every one of the defences that follows is a way of retrying without creating the storm.

The first is to back off, and to do it with jitter. You do not retry immediately and you do not retry at a fixed interval. You wait a little, and if the call fails again you wait longer, and longer again, exponentially, which is the backoff of Part I Chapter 16 and the handshake backoff of Part I Chapter 12, the same idea a third time. This gives the struggling service room to recover instead of being hammered. The jitter matters as much as the backoff. If a thousand callers all backed off by two seconds to the instant, they would all retry at the same instant and recreate the storm, which is the synchronised-expiry stampede of Chapter 8 in a new setting. So each caller randomises its wait, and the retries spread out in time. Backoff gives the service breathing room, and jitter stops the breathing room from ending all at once.

The second is to stop retrying when the service is plainly down. Backoff helps, but a thousand clients politely retrying a dead service still pile up on it. So each caller watches its own success rate against a given service, and when failures cross a threshold it stops calling entirely for a while. It fails fast, without so much as attempting the call, and every so often it lets a single test request through to see whether the service has come back. This is a circuit breaker, and the name is the electrical one. It trips when there is trouble and cuts the connection, protecting the struggling service from the load and the caller from spending time and threads on calls that are going to fail. When the test request succeeds the breaker closes and normal traffic resumes. A thousand services hammering a dead one becomes a thousand services waiting and occasionally checking.

The third is to isolate the failure. When the feed service’s calls to accounts are all hanging, they hold the feed service’s own resources, its event-loop capacity and its connection pool, and if all of those are held waiting on accounts then the feed service cannot serve the requests that never needed accounts in the first place. So you partition the resources, with a limited pool for calls to accounts and a separate one for calls to the cache, so that a hang in one cannot starve the other. The field calls this a bulkhead, after the compartments in a ship’s hull, which stop a breach in one from flooding the whole vessel. One dependency’s failure is confined to the requests that need that dependency, and everything else keeps working.

The fourth is to shed load on purpose. When a service cannot handle the load arriving at it even after all of the above, it faces Part I Chapter 14’s choice. It can fail some requests cleanly and fast, returning “try later”, the 429 of Part I Chapter 20, so that the requests it does accept stay healthy. Or it can try to serve everything and fail all of it by falling over. Load shedding is choosing the first, rejecting a fraction of requests at the door so that the accepted ones succeed. A service that sheds load degrades gracefully, with some users seeing errors and most seeing nothing wrong. A service that does not shed collapses, and everyone gets nothing. This is backpressure from Part I, turned into a survival strategy. It is better to say no to some than to die for all.

Every defence in this chapter is the same realisation seen from a different side. At scale, the reactions to failure are themselves load, and load is what caused the failure. A retry is a request, and so is a health check, and so is a reconnection. When something struggles and ten thousand clients react to it, the reactions can dwarf the original traffic and turn a small problem into a large one. So every reaction has to be damped, by backoff, jitter, breakers, bulkheads, or shedding, so that the system’s response to trouble calms it rather than amplifying it. A well-built system under partial failure gets quieter, and gives the broken part room to recover. A naive one gets louder, and the noise is the outage.

The defences themselves are now things that can be got wrong, and they interact. A circuit breaker set too sensitively trips on ordinary variance and cuts off a healthy service. A timeout shorter than a dependency’s normal latency guarantees failure on every call. Deadline propagation done wrongly makes every request appear to fail at the edge while the services below it succeed to no purpose. And there is a new and humbling category of failure. The system is now so full of interacting safety mechanisms that its behaviour under stress is hard to predict. A retry storm damped in one place can reappear in another, a breaker tripping can shift load onto a neighbour and trip that one too, and the only way to trust the whole is to test it under failure deliberately, rather than hope. So the sixteenth revision of Courtyard adopts timeouts with propagated deadlines, backed-off and jittered retries on idempotent operations, circuit breakers, bulkheads, and load shedding, and accepts in return that the machinery for handling failure is now itself a source of load, and of behaviour that has to be tested by breaking things on purpose.