The Viewer Is 8,000 km Away

Part II Chapter 6 warned you that a bill would arrive, and here it is. Courtyard’s datacentre is in London. The caches from Chapter 8 are doing everything asked of them; the origin barely stirs, the hit rate is ninety-nine percent, and everything in London is fast. Then Anaya, in Sydney, taps the viral video and waits three seconds.

The London cache serves the bytes in under a millisecond. The bytes then travel to Sydney, and Sydney is a quarter of a second away and a quarter of a second back, and that distance is paid on every round trip in the budget from Part II Chapter 6: the lookup, the handshake, the request, the ramp. The sum is three seconds. The cache made the server fast. It did nothing about the distance, because the cache is in London and Anaya is not. What has run out is the speed of light, and it is the one resource in this part that no amount of money replenishes.

You cannot cache your way out of geography by caching in the wrong place, and you cannot buy a shorter round trip; Part II Chapter 5 was clear that latency is the one thing money does not fix. There is one move available, and Part II Chapter 6 named it. Move the bytes closer to Anaya.

Be precise about what moves. You are not moving Courtyard’s servers to Sydney. They need the database, which is in London, and Chapter 2 taught you that state is the hard thing to move. What you are moving is the bytes of the video: a copy, on a machine in Sydney, near Anaya. So you put machines in hundreds of cities, and each holds copies of the content, the videos, the photos, the images and scripts that make up the app itself. When Anaya requests the video she is served from the Sydney machine, over a ten-millisecond round trip instead of two hundred and fifty, and every bar in Part II Chapter 6’s budget shrinks by that factor at once. The whole three seconds becomes a tenth of one.

This network of copies in cities is what the field calls a content delivery network, or CDN, and it is nothing more than the caching of Chapter 8 plus geography: the same cache, the same hit rate, the same questions about staleness, placed near users instead of near the origin. These outer machines, close to the viewers at the edge of the network, are called the edge, and the London datacentre becomes the origin, the place of truth from which the edge caches.

How does Anaya’s request reach the Sydney edge rather than London, when she has done nothing differently? The answer is the anycast of Part II Chapter 7. The edge machines all announce the same address, and the box routes Anaya to the nearest one, because that is what the box does with an anycast address. She types nothing different, looks up the same name, connects to the same address, and the network delivers her to Sydney because Sydney is closer. The primitive you built at the end of Part II is what makes the edge findable. It was always leading here.

Not everything can be served from Sydney, and the line between what can and what cannot is the most important thing in this chapter. Static content can. The video, the photos, the app’s own code and images are the same for everyone. The video Anaya watches is byte for byte the video everyone watches, so a copy in Sydney is as correct as the original in London, forever, since this is the immutable animal of Chapter 2, or until it changes and the invalidation of Chapter 8 catches up. This is what the edge is for, and it is the bulk of the bytes. A video is millions of bytes, and the request that asks for it is a few hundred. Move the millions to the edge and you have moved almost all of the traffic.

Content that is dynamic and personal mostly cannot. Anaya’s feed is hers, assembled fresh from who she follows, out of the database in London. It is not the same for everyone, so there is no shared copy to cache. Her unread message count, her notifications, her drafts, are all specific to her and all require the truth in London, so the request for these still makes the full trip. The edge cannot help, because there is nothing shared to keep a copy of.

So the app splits in two along this line. The shell, meaning the code, the images, and the static structure, loads at once from the edge. The personal data, the feed and the counts, arrives a moment later from the origin, over the long path, ideally on a connection the edge has helped warm up. Anaya sees the app appear immediately and her content fill in afterwards. That two-stage load, which you have seen in every app you use, is this chapter’s line drawn through the product.

There is a middle case worth naming, which is content that is dynamic but shared. The viral video’s view count, a trending list, a public profile: not personal, but changing. The edge can cache these, with a short lifetime, accepting the staleness of Chapter 8, since a view count thirty seconds old at the edge is fine. This is where the edge earns the most and where the invalidation problem is hardest, because now the stale copies are in hundreds of cities.

You have put copies in three hundred cities, and now the viral video’s copy expires, at the lifetime from Chapter 8, at the edge this time. It expires at roughly the same moment in all three hundred cities, because they all cached it at about the time it went viral. Three hundred edges miss at once, and each does what a cache does on a miss and fetches from the origin. Three hundred requests reach London together. Even if each edge is serving thousands of viewers and coalescing locally, that is still three hundred simultaneous fetches from the origin for one file. And the origin was sized for the edge’s hit rate, which was Chapter 8’s final warning. It normally sees almost no traffic, because the edge absorbs it, and it is not provisioned for three hundred cities all missing together. This is Chapter 8’s stampede at global scale, with the origin far away and small.

The fix is one more layer of caching. Between the three hundred edges and the origin you put a small number of larger caches, one per region, say. The edges do not fetch from the origin; they fetch from their regional cache. When three hundred edges miss they reach a handful of these, and each one, using the coalescing of Chapter 8, collapses its region’s many misses into one fetch to the origin. The origin sees a handful of requests instead of three hundred. This middle tier is called a shield, and it is Chapter 8’s coalescing applied across geography: fan the requests in, not out, before they reach the scarce thing at the centre.

Staleness is now geographic. A copy in Sydney can be stale independently of a copy in São Paulo, and invalidating a changed file means reaching three hundred cities, some of which are slow to hear, so that for a while the world shows a mixture of old and new. It is the TTL map of Part II Chapter 4, except that every point on it is a city holding content. When a friend says the video looks different on her phone, there is a new answer: you are on different edges, holding copies of different ages.

The edge also now stands between every user and Courtyard, which makes the edge provider a dependency Courtyard cannot function without. If the edge network has a bad day, Courtyard has a bad day, everywhere, and it is not Courtyard’s outage to fix. You have traded latency for a dependency on someone else’s global network. The trade is almost always worth making, and it is never free.

So the speed of light ran out, and you pushed static content to edge caches in many cities, found by anycast and shielded from the origin by a regional tier. Staleness is now geographic and slow to invalidate across the world, content that is personal still pays the full distance, and the edge provider is now a hard dependency. Static content is close to its viewers. Everything else still crosses the ocean.