Many Front Doors

Anaya lives in Sydney, and she sends Mira a message. Chapter 9 put Courtyard’s static content near everyone, so the shell of Anaya’s app arrives from an edge a few milliseconds away. But the message itself is a write, and a write cannot be served from a cache, because it changes the truth, and the truth is in London. So the message crosses to London to be stored, and the acknowledgement crosses back. The same is true of every post, every like, and every follow from every user on Earth. The static content was the easy half, the shared and unchanging bytes. Every write still goes to one datacentre.

That leaves two problems, the second worse than the first. The first is latency, once more. Anaya’s writes pay the full round trip of Part II Chapter 6, every time. Her feed loads its shell instantly from the Sydney edge and then waits a quarter of a second for every actual interaction to reach London and return. Chat, which is all writes and all interactive, is slow for her in a way that no edge cache can fix.

The second problem is that London is one datacentre. Chapter 5 made the database inside it survive the loss of a machine, and the loss of a switch. It cannot survive the loss of London itself: a power event across the whole facility, a fibre cut that isolates the city, a fire, a flood. When London goes, Courtyard goes, for everyone, everywhere. Sydney and São Paulo go dark with it, because the truth lived in one building and the building is unreachable. Every piece of redundancy you built in Chapters 3 through 6 was inside one datacentre, and at this scale a datacentre is a single point of failure, the largest one you have left. One region is what has run out.

So you put a whole Courtyard in a second region. Say it is Sydney, with application servers, a connection tier, a database, and caches. Now Anaya’s writes go to Sydney, ten milliseconds away, rather than to London. And if London burns, Sydney carries the world. You have two full regions, each able to serve the users near it, each a backup for the other.

You route users to the nearest region the way you have routed everything else, with Part II Chapter 7’s anycast, or with geographic DNS, which is the lever from Part II Chapter 4 pointed at different addresses according to where the asker is. Anaya gets Sydney and Mira gets London. Each writes locally, and each is fast.

Most of the system divides without complaint. The application tier is stateless, as Chapter 2 made it, so a copy in each region is trivial. The edge and the caches are copies by nature, from Chapters 8 and 9, so they divide too. Everything divides cleanly except the one thing that never does, the thing Chapter 2 warned would be the hard part every single time, which is the database, where the truth lives. There are now two truths, on two continents, and they are going to disagree.

Anaya in Sydney and Mira in London are in the same group chat. Anaya sends a message, and it is written to the Sydney database. Mira sends one at very nearly the same moment, and it is written to the London database. Each region has to tell the other, so each message crosses the ocean, and the crossing takes a quarter of a second each way, the floor from Part II Chapter 6 that no engineering can lower.

For that quarter of a second, Sydney has Anaya’s message and not Mira’s, and London has Mira’s and not Anaya’s. A third member of the group, in Sydney, sees Anaya’s message first. A fourth, in London, sees Mira’s first. When the cross-region writes arrive, each region appends the other’s message, and now the two regions hold the same two messages in different orders. Anaya’s group shows hers and then Mira’s; Mira’s shows hers and then Anaya’s. Neither is wrong. There is no fact of the matter about which message really came first, because they happened a quarter of a second apart on two continents, and the speed of light will not let either region learn of both in time to order them.

This is staleness in its final and worst form. Chapter 3’s replica lag was a copy running behind the truth, which was a nuisance, but there was one truth and the copy would catch up. What you have now is two copies that are each the truth for their own region, that have diverged, and that will never converge on their own, because there is no single truth for them to converge to. It is Chapter 5’s partition choice, except that the partition is permanent, because it is an ocean, and both sides are always accepting writes.

There are three ways to resolve this, and every large system uses one of them or a mixture. None of them is free.

The first is to pin ownership. You decide, for each piece of data, that one region owns it. This group chat is owned by London, so all writes to it go to London, wherever the writer happens to be. Anaya in Sydney sends a message, it is routed to London to be written, and the acknowledgement comes back. There is one truth for this group, London’s, so there is no conflict, ever. The cost is that Anaya’s writes to a London-owned group pay the cross-region round trip. But most data has a natural home. A group is mostly used by people in one region, and a user’s own profile is owned by their home region, so most writes are local and only the cross-region cases pay. This turns an impossible problem, resolving conflicts, into a solvable one, routing each write to its owner. It is what most systems choose, because no conflicts with an occasional slow write is a better bargain than always fast and sometimes contradictory.

The second is to accept the conflict and merge. You let both regions write, accept that they will disagree, and define rules for reconciling them. For a group chat the rule might be to order messages by timestamp, and when timestamps tie, by some fixed tiebreak, so that both regions independently arrive at the same order once they have seen both messages. The orders converge because the merge rule is deterministic. For a like count, you merge by adding up each region’s increments. For a field where the last edit wins, you keep the one with the later timestamp. This keeps every write fast and local, at the cost of designing a merge rule for every kind of data, and some data has no good merge rule. When two people edit the same sentence, last write wins silently discards one person’s work. Data types that merge cleanly by construction exist and are used for precisely this, but they do not cover everything.

The third is to pay the round trip. For the data that must be both consistent and global, such as account balances if Courtyard ever handles money, or anything for which two truths would be unacceptable, you make every write coordinate across the regions before it is confirmed, so that there is one agreed truth. This is Chapter 5’s synchronous choice at the scale of an ocean. Every such write pays at least one cross-region round trip, because it is not confirmed until the regions agree. It is correct and it is slow, and you reserve it for the rare data that truly needs it.

This is the chapter at which a great many systems are over-engineered, so a word about proportion is due. Most services do not need multiple regions accepting writes. A great many run in one region with a warm standby in another. The standby replicates asynchronously, as in Chapter 3, and stands ready to be promoted if the primary region fails, at the cost of some lost data and some downtime during the failover. That is far simpler than true multi-region writes. It removes the conflict problem entirely, since only one region writes at a time, and it survives the loss of a region with a few minutes of disruption rather than none. For almost everyone it is the right answer.

True active-active multi-region, with both regions writing at once, conflicts and all, is what you reach for when you need writes to be fast on several continents and also need to survive the loss of a region with no downtime whatever. That is a real need for a global chat product like Courtyard, which is why this chapter exists, but the club is smaller than architecture diagrams suggest. The honest default is one region and a standby, and the three answers above are for when you have confirmed that you need more.

Several things can now go wrong. Regions can diverge, and reconciling them can silently discard writes, as when last write wins throws away one editor’s change. Cross-region replication can lag by far more than replication within a region, since an ocean is long and congested, so the window of disagreement is seconds rather than milliseconds. A region failover can lose the writes that had not yet crossed. Routing itself becomes a source of bugs, because a user who travels from London to Sydney might be routed to the wrong region for their data and read their own writes stale across an ocean, which is Chapter 3’s read-your-own-writes problem grown to continental size. And the operational surface more than doubles, with two of everything, kept in sync, deployed in sync or carefully out of sync, monitored in both places, with a whole new class of incident in which the regions disagree, an incident no single-region system can have.

That is the twelfth revision of Courtyard. One region ran out, and you deployed full regional datacentres with users routed to the nearest. The databases now diverge, and they are reconciled by pinning ownership, by merge rules, or by cross-region coordination, with each write assigned to one of the three according to what its data can tolerate. Most data is pinned to a home region, so most writes stay local. Courtyard now spans continents, and the system is not the only thing that has grown.