Everyone Can Read It

Go back to Chapter 4 for a moment. Theo spliced a tap into the wire, and for a while every frame you sent Mira arrived at his card as well. The switch mostly put an end to that, since your frames are no longer on his wire, but “mostly” is carrying a good deal of weight in that sentence. Your frames are still on your own wire, and on the switch, and on every wire and box between you and Mira once she is across the city. Anyone with access to any of those can put a card into the promiscuous mode of Chapter 4 and read everything that passes.

Before following Theo, I should confess something about this chapter. In every chapter so far you have invented the fix yourself. In this one you cannot, or at least not the core of it. The mathematics that lets two strangers agree on a secret over a wire an attacker is reading is hard to derive, and nobody sits down in a workshop and reinvents it. So you will work out the wants, and know precisely what you need and why, and then the mechanisms that satisfy them will be shown to you rather than built. The effort will go where it matters for someone building things, which is understanding what the machinery promises, where it bottoms out, and how it fails. There are four places in this book where the centre of a chapter has to be handed over rather than built, and this is the first of them.

So let Theo read. This is what he sees when you message Mira:

POST /conversations/mira/messages HTTP/1.1
Host: courtyard.example

{ "text": "the spare key is under the third pot", "idempotency_key": "..." }

He sees everything. The text, in full. And in a moment, when Courtyard has a notion of accounts, he will see the password you typed to log in and the token that proves you are you. All of it crosses in plain characters, readable by anyone on the path. Every agreement you have built assumed that the participants were you and Mira. None of them allowed for a third party reading along, and by default there is one, and there always was.

Three different things are wrong here, and they need separating, because they need three different fixes. The first is secrecy. The eavesdropper can read the message, and you want him not to be able to; whatever crosses the wire should be meaningless to anyone but Mira. The second is integrity, and it is worse than reading, because the attacker can change the message. He is on the path, so he can drop your frame and inject his own. the spare key is under the third pot becomes the spare key is under the front mat, and Chapter 3’s check is no help, because the attacker recomputes it over the altered message. That check was designed to catch a wire’s random flips, not a person’s deliberate edit. You want to know if a single byte was changed by anyone but the sender.

The third is identity, and this is the one that turns out to be hard. Suppose you fix secrecy: you and Mira share a secret and scramble everything with it. How did you agree on the secret? You had to exchange something over the wire to establish it, and the attacker saw that exchange. If he could take part in it, pretending to be Mira to you and to be you to Mira, then you would share a secret with the attacker while believing it was Mira, and you would scramble every message perfectly so that only the attacker could read it. That is secrecy against everyone except the person you did not know you were talking to.

That third want is the whole game. Secrecy and integrity are comparatively easy once you and Mira share a secret. Getting a secret that only you and Mira share, over a wire a liar is sitting on, is the hard part, and it is the reason this chapter cannot be derived.

Now for the piece you are shown rather than build. There is a kind of mathematics, and this is the part to take on faith or to study elsewhere, in which you and Mira each pick a private number, exchange the results of a one-way calculation on those numbers in the clear, and each combine what you received with your own private number to arrive at the same shared secret. An eavesdropper who saw both public halves cannot compute it. The operation is easy to do and effectively impossible to undo, so the public halves reveal nothing usable. You and Mira end up holding an identical secret that never crossed the wire.

That is key exchange, and it solves secrecy. With a shared secret you can encrypt: scramble each message with the secret so that it is meaningless without it, and attach a tag, computed with a keyed version of Chapter 3’s check, that only someone holding the secret could have produced, which catches any alteration. Secrecy and integrity both follow from one shared secret. Take those as given. What key exchange does not solve is the third want. You have agreed on a secret with whoever was on the other end of the exchange, and nothing so far proves that it was Mira.

This is the want you can reason about, so reason about it. You need to know that the public half you received came from Mira and not from an attacker impersonating her. You have never met Mira’s machine and have nothing of hers to compare against, so you cannot verify her directly. The only way this ever works is the way it works in the physical world when you cannot personally verify someone: somebody you already trust vouches for them.

Suppose there is a party, call them an authority, that you already trust, because their vouching credential was built into your machine before you ever connected to anything. Mira goes to this authority, proves she controls courtyard.example, and the authority issues her a signed statement saying that the holder of this particular key is courtyard.example and that the authority vouches for it. The signature is something only the authority could produce and anyone can check.

Now, when you connect, Mira sends you that signed statement along with her public half. You check the signature against the authority’s credential you already have. If it verifies, you know that the authority, whom you trust, says this key belongs to courtyard.example. The attacker cannot forge the statement, because he cannot produce the authority’s signature. He can replay Mira’s statement, but it is bound to Mira’s key and not to his, so it does him no good; he does not have the private number that matches.

The signed statement is a certificate, and the authority is a certificate authority. The thing that makes the whole arrangement stand, or fall, is that last step. You trusted the authority because its credential was already on your machine.

It cannot be authorities all the way down, so look at where it stops. Your machine shipped with a list of root certificates, a few dozen to a few hundred of them, belonging to organisations that the makers of your operating system and browser decided to trust. That list is the foundation, and everything else chains to it. Mira’s certificate is usually not signed by a root directly. It is signed by an intermediate authority, whose own certificate is signed by a root. So when you connect, Mira sends a chain: her certificate, signed by an intermediate, and the intermediate’s certificate, signed by a root. You verify each link, Mira’s against the intermediate and the intermediate against the root, until you reach a root that is on your built-in list. If every link checks and the chain ends at a root you already trust, you are satisfied. If the chain ends at a root you have never heard of, you refuse.

That is the padlock in your browser. It does not mean that the site is safe or that the company is honest. It means, precisely, that a certificate authority on your machine’s trusted list vouches that the key you are exchanging secrets with belongs to the name in the address bar, and that the certificate has not expired or been revoked. It means nothing more. A phishing site can hold a perfectly valid certificate for its own deceptive name. The padlock verifies who, not whether they deserve it.

You will spend far more of your life diagnosing this machinery than implementing it, so it is worth spending some effort on the ways it fails. The commonest is that the certificate has expired. Certificates have end dates, a year or two out, so that a stolen key cannot be used to impersonate forever and so that revocation lists stay short. When one lapses, every client refuses the connection, and a service goes dark, not because anything was compromised but because someone forgot to renew. It is one of the most common self-inflicted outages there is, and the fix, automatic renewal, is now standard because humans forgot too often.

The next is that the name does not match. The certificate is for courtyard.example and you connected to www.courtyard.example, or to the raw address. The certificate is valid and the name is wrong, so the connection is refused, and correctly so, since a certificate for one name proves nothing about another. Then there is the incomplete chain. Mira’s server sends her certificate but forgets to send the intermediate. Your machine has the root but not the intermediate, so it cannot connect the two, and the chain is broken even though every certificate in it is genuine. Browsers sometimes paper over this by fetching the missing link; other clients do not, which is why a site can work in a browser and fail from a program.

A subtler failure is a wrong clock. Verifying that a certificate has not expired requires knowing the current time. A machine whose clock is badly off, a freshly booted device or a server with a dead battery, will reject valid certificates as expired or accept expired ones as valid. Time is a security dependency, which is not obvious until it bites. And the gravest failure is a betrayed root. The whole system rests on the authorities behaving. An authority that issues a certificate for a name to the wrong party, whether tricked, coerced, or breached, can enable the very machine-in-the-middle attack this chapter is about, complete with a valid padlock. This has happened. The defences, public logs of every certificate issued so that a wrongly issued one can be spotted, are patches on a trust model that is, at bottom, a list of organisations you were told to trust.

One design question you can reason about is where in the tower all of this goes. It sits above the transport and below the application. It goes above the transport because it needs the reliable, ordered stream of Chapter 11; it is exchanging a structured handshake and cannot cope with the transport’s raw losses itself. It goes below the application because everything the application sends, every verb, path, header, and body from Chapters 18 through 20, should be inside the secrecy rather than outside it.

Put it there and consider what each layer can still see. Someone reading the wire sees the transport headers, which is to say the four-tuple, so he knows which two machines and ports are talking and roughly how much, but the entire application payload is opaque to him. He knows you are talking to courtyard.example. He does not know what you said. That residue, who is talking to whom and how much, is called metadata, and it survives encryption because the boxes in the middle need the addressing to do their job. It is the part even a perfect secure channel cannot hide, and in Part II you will see that it leaks the destination name too, at least until a further layer of the tower learns to hide even that. The name of the whole construction, key exchange followed by encryption and keyed integrity, riding above the transport, is TLS, and the s in https is this chapter.

Courtyard’s messages are now unreadable and unalterable by anyone on the path, and your client refuses to talk to anything that cannot prove, through a chain ending in its own trusted list, that it is really Courtyard. Theo’s tap gets him nothing but the knowledge that you and Mira are talking.

You have reached the top of the tower. Every rung is built: bits, framing, integrity, identity, location, resolution, ports, conversations, the loop, segmentation, reliability, connections, liveness, flow, congestion, transports, protocols, evolution, retries, and now security. Before you go anywhere new, it is worth climbing back down through all of it once, with everything in place. That is the next chapter, and the one after it is about why the whole thing is slower than it should be.