Light Is Slow

Mira has a friend in Sydney. Her name is Anaya, and one evening she opens Courtyard for the first time. Her connection is good. Mira’s server, in London, is fast and idle. Anaya taps a photo, and three seconds pass before it appears. Three seconds is long enough to wonder whether the app is broken, and almost none of that time is spent on the photo.

Let’s start with the part that cannot be changed. Light in glass travels at about two hundred thousand kilometres a second, which is two-thirds of its speed in a vacuum. London to Sydney, by the cables that actually exist, is around seventeen thousand kilometres. So a piece takes at least eighty-five milliseconds to get there, and a round trip, there and back, takes at least a hundred and seventy. That is before any box has held the piece for an instant, before any queue, before anything at all except the glass. The real figure is higher, because cables do not run straight and the interlude after Chapter 2 routed you through places you did not choose. Two hundred and fifty milliseconds is a good working number for that round trip.

A quarter of a second per round trip, then, at the very least. No plan, no hardware, and no money changes it. It is the length of the pipe from Chapter 5, and this pipe is as long as pipes get. What remains is to count the round trips, so let’s follow Anaya’s tap through the tower and note every time her machine has to wait for London to answer before it can take the next step.

First her machine must look up the name feed.courtyard.example, as in Chapter 4. It asks its resolver. If the resolver has the answer cached, that is quick and local. If not, the resolver walks the tree, and one or more of the authorities it consults may be far away. Call it one round trip, if you are lucky.

Then it must open the connection, with the handshake of Chapter 12. Her machine sends the first message, London replies, and her machine confirms. Data can ride on the third message, but nothing can be sent until the second has arrived. That is one more round trip, and the total stands at two.

Then it must secure the connection, with the key exchange and certificate check of Chapter 21. In its current form that is one more exchange: send the opening, get back the certificate and the key material, verify, and the channel is secure. One round trip, and the total is three.

Only now does the request itself go out: GET /users/mira/photos/42. London receives it, finds the photo, and starts sending. The first byte of the response arrives after one more round trip, and the total is four. Four round trips, a full second, before a single byte of the photo has arrived, and every one of them was conversation between the two ends rather than transfer.

Now the photo itself, which is three megabytes. Chapter 16 tells us what happens: the connection is new, so it starts slow, sending about fourteen kilobytes in the first round trip and doubling each trip while nothing is lost. Fourteen, twenty-eight, fifty-six, a hundred and twelve, two hundred and twenty-four, four hundred and forty-eight, nine hundred, eighteen hundred, thirty-six hundred kilobytes. That is eight or nine round trips to get three megabytes through, even though the pipe is wide enough to carry it all in a fraction of one. The pipe is not the constraint. The ramp is.

Thirteen round trips, at a quarter of a second each, comes to just over three seconds. There is Anaya’s wait, accounted for to the millisecond, and what strikes me most is the proportion. The bytes of the photo, at the bandwidth available, would take a couple of hundred milliseconds to move. Everything else, well over two and a half seconds, is the two machines talking about moving them.

Every one of those bars is a target, and we can go through them in turn.

The first thing to do is reuse the connection. This is the debt from Chapter 16, and it is the biggest single win. The handshake and the security exchange, two round trips together, happen once per connection, not once per request. Keep the connection open after the first photo, and the second photo skips both. So does the third, and the profile, and the next twenty things Anaya taps. Her second request costs one round trip instead of three, and it arrives on a connection whose slow-start window is already warm, so the ramp is shorter too. Every serious client keeps a pool of open connections for this reason, and the most common performance mistake in programs that call other programs is opening a fresh connection every time.

The second is to fold the handshakes together. The transport handshake and the security handshake are two conversations that could be one. Newer versions of the security layer already cut it to one round trip. The transport from Chapter 17’s fork goes further and merges the two: connection and security are established in a single exchange, one round trip instead of two on every fresh connection.

The third is to resume rather than restart. Anaya talked to London yesterday, and both sides remember the secret they agreed on. So today, instead of a full key exchange, her machine can say, in effect, let’s continue from last time, and send data in the very first message, with zero round trips of security setup. There is a subtle cost. That first message is sent before the server has confirmed anything, so an attacker could record it and replay it, and only requests that are safe to repeat, the idempotent ones of Chapter 20, may ride on it. The rungs connect.

The fourth is to cache the lookup. The name does not change often, and Chapter 4 gave it a lifetime. After the first lookup, Anaya’s machine and her resolver both remember the answer, and that bar vanishes for the rest of the day.

The fifth is to send less, so that the ramp is shorter. A three-megabyte photo is nine round trips of slow start. The same photo, compressed properly and sized for a phone screen, might be three hundred kilobytes, which is five round trips. The pipe did not change. You asked it to carry less, and the doubling reached the total sooner.

The sixth is to ask for things at once rather than in turn. If the page Anaya is looking at needs the photo and a profile and a list of comments, fetching them one after another costs three request round trips in a row. Fetch all three at once, on the one warm connection, and you pay one round trip for all of them.

Do all of that and Anaya’s second photo costs about one round trip plus a short ramp. Half a second, say, down from three. That is a real improvement, and it is what a competent engineer does with Part I’s tower on a long path.

And then you are done, because the round trip is still a quarter of a second, and there is still at least one of them. The request has to reach London and the response has to come back. Every technique above removed trips; none of them shortened one. You have stripped the conversation to its minimum and the minimum is a quarter of a second, and it will be a quarter of a second for as long as Anaya is in Sydney and the photo is in London.

There is one move left, and it is not an optimisation. It is a change of geography. Put the photo in Sydney. Not the server, but the photo: a copy of it, on a machine near Anaya, so that her request travels ten kilometres instead of seventeen thousand and the round trip is ten milliseconds instead of two hundred and fifty. Then every bar in the budget shrinks by a factor of twenty-five at once, and the whole sequence of lookup, handshake, request and ramp fits inside the time one London round trip used to take.

You have just described the central idea of Part III, and its central enemy. The enemy is distance. It cannot be beaten; it can only be avoided, by not being far. Everything Part III builds, the copies of data near users, the servers in many regions, the layers that decide which copy is closest, exists because of the arithmetic in this chapter. Chapter 4 handed you the lever that points users at a machine. This chapter tells you why you will want to point different users at different machines: the one that is near is the one that is fast, and nothing else is.

You can now account for every millisecond between Anaya’s tap and her photo, and you can name which technique removes which round trip, and you know where the accounting stops. That is the whole of performance work on a long path: count the trips, remove the ones you can, and then move. One chapter remains in this part, and it concerns what happens when the path does not merely get long but breaks.