The Team Ran Out

Courtyard is one codebase. It started small and it grew, and now forty engineers work on it: the chat team, the feed team, the media team, the accounts team, the search team, and the notifications team, all committing to one repository, building one program, and deploying one artifact. Every chapter so far has run a machine out of something, a processor, a disk, a link, a pool of connections, a region. In this one every machine is fine. It is the people who have run out of something, and what they have run out of is coordination, which I think is the most underestimated scaling limit there is.

The feed team finishes a feature and wants to ship it. To ship, they have to deploy the whole codebase, because it is one program, which means they would also be shipping the chat team’s half-finished work. So they cannot. They wait until everyone’s changes are ready, or they coordinate a careful merge. The deploy itself restarts everything, chat, feed, media, and the rest, so a feed deploy is also a chat deploy and the chat team has to be watching. A bug in the accounts team’s code crashes the process that also runs the feed, so the feed team’s uptime depends on the quality of the accounts team’s code review. Two teams need the same file changed in incompatible ways and have to negotiate. The test suite takes forty minutes because it tests everything, and every team runs all of it for every change.

None of this is a technical limit. What has saturated is the ability of the humans to keep out of each other’s way. Forty people editing one program, deploying one artifact, and sharing one failure domain is a coordination load that grows faster than the team, because every new engineer adds not one worker but a new set of interactions with everyone already there. The codebase scaled. The organisation around it did not, and that is what ran out.

The fix mirrors every other fix in this part, which is to stop sharing. This time, though, you are not dividing data or load. You are dividing the codebase, along the lines of who owns what.

Give each team its own program, its own repository, and its own deploy. The chat service is one program, which the chat team owns and deploys whenever they like. The feed service is another. There is a media service, an accounts service, and so on, and the six teams, splitting their work along its natural seams, end up with a dozen or so services between them. Each runs on its own machines, is deployed on its own schedule, has its own failure domain, and is tested on its own. The feed team ships ten times a day without asking anyone. A crash in accounts takes down accounts, not the feed. Each team’s test suite covers their own service and runs in three minutes.

What you have made are services, or microservices when they are small and numerous, and the thing to understand about them, the thing this chapter exists to say, is that the reason to split is organisational rather than technical. You do not break Courtyard into services because it makes the software faster. It makes the software slower, as you are about to see. You do it because it lets forty engineers work without blocking one another. Services are a solution to a people problem, and their boundaries should follow the teams, which in turn follow the product’s natural divisions. Draw the boundaries anywhere else, by technical layer rather than by ownership, say, so that every feature needs changes across five services and three teams, and you have kept all the costs and lost the one benefit.

Now for the cost, which is steep, and which amounts to the whole back half of this book arriving at once. Yesterday, when the feed service needed a user’s display name, it called a function, getUser(id). The call was instant and it never failed, because it ran in the same process and the same memory. Today the feed team’s service and the accounts team’s service are different programs on different machines, and getUser(id) is a network call. The feed service sends a request across the network to the accounts service and waits for a response.

A function call has become a network call, and a network call is bound by Part II’s contract. It can be slow. It can be lost. The accounts service can be up while the network to it is down. The request can arrive twice, and it can half-complete. Everything that Part I spent twenty-three chapters building, timeouts, retries, idempotency, ordering, backpressure, is now required inside Courtyard, at every boundary between two services, for what used to be a function call that could not fail.

So for its call to accounts, the feed service now needs a timeout, from Part I Chapter 11, and with it a decision about how long to wait and whether, on giving up, to fail the whole feed or to show it without the name. It needs a retry policy, from Chapter 20, and with it the question of whether getUser is idempotent. It is a read, so it is safe to retry, but a call to postMessage is not, and needs an idempotency key. It needs a plan for when accounts is simply down, whether to fail, or use a cached name, or show a placeholder. And it needs the discipline of Part I Chapter 15, because a slow accounts service must not block the feed service’s event loop. Every arrow between two services in your architecture diagram is a full instance of Part II’s unreliable box, and every one of them needs the whole of Part I’s tower to survive it.

There is one thing you lose outright, and that is the transaction across modules. In one program with one database, creating a user and adding them to the default group was a single transaction. Both happened or neither did, and the database guaranteed it. Across an accounts service and a groups service it is two network calls to two databases, and the second can fail after the first has succeeded, leaving a user in no group at all. No transaction spans two services. Anything that used to be atomic across modules is now Chapter 4’s half-completed cross-shard operation, everywhere, and coping with it is Chapter 15’s job.

There is a way of doing this chapter that is worse than not doing it at all, and it has a name, the distributed monolith. You split into services, but you split them wrong, not along lines of ownership but arbitrarily, so that the services are deeply entangled. Every feature still requires changing five services together. They all share one database, so they are not really independent, and a schema change breaks all of them, which is Chapter 14’s problem. They must be deployed together, in a particular order. Service A cannot function if service B is down, and vice versa, so they fail as a unit anyway.

That is the worst of both worlds. You still cannot deploy independently, which was the whole point, because the services are coupled. And you have added the entire network-failure bill, the latency of all those calls, and the operational complexity of running twenty programs instead of one. You paid every cost of services and got none of the benefit, because the benefit comes only from independence, and you split without achieving it. This is the most common way service architectures fail. It is why the question of whether to adopt microservices is answered with “not until you know where the boundaries are”, and you usually only know that after the monolith has taught you where the teams and the seams actually lie.

The honest guidance follows from this, and it is to start with one program. A single codebase is faster to build and easier to change, and it carries none of this bill. For a small team it is the correct choice, because the coordination cost that justifies splitting does not exist yet at five engineers. You split when the team has grown enough that coordination is the bottleneck, and you split along the seams the monolith has revealed. Courtyard earned its way to services by first being a monolith that forty people could not share. A system that starts life as fifty microservices built by five people has bought a solution to a problem it does not have, at ruinous cost.

Everything Part I and Part II described can now fail internally, at every service boundary, multiplied by the number of boundaries. A request that used to be one program is now a call graph across a dozen services. It succeeds only if enough of them succeed, it is as slow as the slowest path through them, which is Chapter 4’s tail latency spread across services, and any one of them being down degrades or breaks it. A schema shared between two services couples them silently. And when something is slow, the cause could be any service in the graph or any network hop between them, which is a problem large enough to have its own chapter, because you can no longer even see what happened.

That is the thirteenth revision of Courtyard. Coordination ran out, and you split the monolith into services along team lines so that teams ship independently. Every call between services is now an unreliable network call that needs timeouts, retries, and idempotency; operations that cross services are no longer atomic; and a badly drawn split yields a distributed monolith with all of the costs and none of the independence. The seams, meanwhile, turn out to be less clean than the diagram suggests.