More Readers Than the Disk Can Serve

The database has its own machine now, with a fast disk, and it is the busiest machine Courtyard has. Count what hits it. Every message sent is one write. Every message read, by every member of the conversation, every time they open it, is a read. Every profile view, every feed refresh, every check of how many likes something has is a read. The ratio comes out somewhere around a hundred reads for every write, and the reads alone are enough to saturate the disk. What has run out is not the disk’s capacity but its ability to serve readers.

You could buy a bigger database machine, as Chapter 1 advised, and you should, once. But the database is the one component where the cost curve bites hardest, because the biggest database machines are the most expensive machines there are, and you are going to want that headroom later for a problem copies cannot solve. So look at the shape of the load instead. It is reads, and reads do not change anything, and a thing that does not change anything can be answered by a copy.

You set up a second database machine. You tell the first, which we will call the primary, to keep doing what it does, but also to ship every change it makes, in order, to the second. The second, the replica, applies each change as it arrives, so that it holds the same data as the primary, a moment behind.

Now you split the traffic. Writes go to the primary; they have to, since it is the one that ships changes. Reads go to the replica. The primary’s load drops by whatever fraction you route away, which given the ratio is most of it. If you need more read capacity you add a second replica, and a third. Each takes a share of the reads, and the primary ships its stream of changes to all of them.

The shipping deserves a second look. It is a sequence of changes, in order, that must arrive complete, which is to say it is Part I Chapter 11’s reliable stream. Replication is, at bottom, one machine sending another an ordered log of everything it did, over a connection. The tower is inside the database now.

And it works. Read throughput multiplies and Courtyard is fast again. Revision three looks easy to write, and then Mira uploads a photo.

She taps upload. Her client sends the photo to the application, which writes a record to the primary: photo 91, owner Mira. The primary acknowledges. The client, pleased, immediately fetches her profile page to show her the new photo. That is a read, and reads go to the replica.

The replica has not applied that write yet. It is a hundred and fifty milliseconds behind the primary, which is the time it takes for the change to cross the wire and be written to the replica’s disk. It looks up Mira’s photos and returns the list it has, which does not include 91. Mira’s profile shows no new photo. She frowns and taps upload again.

If Part I Chapter 20’s idempotency key is in place, the second upload is recognised as a repeat and nothing duplicates. If it is not, she now has two copies of the same photo, and both will appear a moment later, when the replica catches up, and she will wonder what happened. Either way, Courtyard has just told a user that her own action did not happen. That is a bug that did not exist yesterday. You introduced it, on purpose, in exchange for read throughput.

What you are looking at is called replication lag, and it is the antagonist of Part III walking on stage. You have met it twice before: in Part I Chapter 6’s identity cache that pointed at a dead card, and in Part II Chapter 4’s name records that half the world remembered wrong for a day. It is the same thing each time. A copy sits behind the truth by some amount of time, and anyone reading the copy during that time gets the past.

There is a difference from those two occasions, though, and it is an uncomfortable one. The name record’s staleness was bounded and chosen; Mira set the lifetime. The replica’s lag is neither. It is whatever it happens to be: a hundred milliseconds on a quiet afternoon, several seconds during a burst of writes, minutes if the replica’s disk is slower than the primary’s and it falls behind and cannot catch up. You do not set it. You observe it. And a replica that is falling further behind is Part I Chapter 14’s slow consumer, with the primary producing changes faster than the replica applies them, a queue growing, and the same three choices about what to do.

To see where the trade lives, look at the moment when the primary says done to the client. The primary could wait. It could receive the write, ship it to the replica, wait for the replica to confirm it has applied it, and only then acknowledge the client. Now the replica is never behind at the moment of acknowledgement, and a read that follows a write will find it. This is synchronous replication. Its cost is that every write pays a network round trip plus the replica’s disk before it is confirmed, so writes get slower. And if the replica is down, or the network to it is cut, writes stop, because the primary is waiting for a confirmation that will not come. You have coupled the primary’s availability to the replica’s.

Or the primary could decline to wait. It acknowledges the client as soon as its own disk has the write and ships the change to the replica afterward, in the background. Writes are fast, and the replica being down does not stop anything. This is asynchronous replication, and its cost is the lag, plus one thing worse. If the primary’s disk dies after acknowledging but before shipping, that write is gone. The client was told it happened, and it did not survive.

There is no free option. Synchronous replication trades write latency and availability for freshness and durability, and asynchronous trades the reverse. Most systems choose asynchronous for most replicas and accept the lag, because slow writes are felt by every user on every message and lag is felt only by the user who just wrote. But choose is the operative word, and you will see the choice again in Chapter 5, with the stakes raised.

You cannot make the lag zero without going synchronous. What you can do is stop it lying to Mira, and there are three ways to go about it.

The first is to let a user read their own writes. The bug is specifically that Mira wrote and then read and did not see it. Other people not seeing her photo for a hundred and fifty milliseconds is invisible, since they did not know it existed. So after a user writes, you route that user’s reads to the primary for a short while, a few seconds, longer than any plausible lag. Everyone else’s reads still go to replicas. The primary takes a trickle of extra reads from recent writers, and nobody sees their own action vanish. There is a sharper version. The primary’s log has positions, so when it acknowledges a write it can tell the client that this was change number 88,102. The client passes that number along with its next read, and the application sends the read to any replica that has applied at least 88,102, or waits for one that has. That is precise, and it involves no time-based guess.

The second is not to let time run backwards. Suppose two replicas, one at change 88,102 and one at 88,050. Mira’s first read goes to the fresher one and she sees her photo. Her second read, a moment later, goes to the staler one and the photo is gone. Then it is back. A user who sees a thing appear, disappear and reappear will not trust anything you show them. The fix is to keep a user on one replica for a session, so that whatever staleness they see at least moves in one direction. The field calls this monotonic reads, and it costs you a little flexibility in spreading load.

The third is to classify the reads, because not every read needs the same freshness. Mira’s own profile, right after she posts, must be fresh or she doubts the system. The feed of other people’s photos can be a second stale and nobody can tell. A like count can be a few seconds stale. The balance of an account, if Courtyard ever has payments, can never be stale, and goes to the primary only. This is Part I Chapter 20’s move, classifying the request by what a wrong answer costs, applied to reads instead of retries. Most reads tolerate lag. Route those to replicas without a second thought, and spend the primary’s scarce read capacity only on the ones that do not.

Yesterday there was one truth. Today there are several copies of it, and at any instant they disagree by a little. Every read from a copy might be stale, by an amount that depends on load, and the staleness can show a user that their own action failed when it did not.

There is a subtler thing waiting as well. The replica holds a full copy of the data, so if the primary dies, the natural move is to promote the replica: tell it to start accepting writes, and point the application at it. That is the whole point of having it. But which writes did it have? Under asynchronous replication the last few acknowledged writes may not have reached it, and they are lost. And if the primary is not actually dead, only unreachable from where you are standing, you now have two machines accepting writes for the same data. That is Chapter 5’s subject. Revision three of the design document records it: read throughput ran out; you added replicas that serve reads from a lagging copy; a user can be shown a past that contradicts their own action, and promoting a replica can lose writes or create two primaries.