Three-Way Introductions

Every piece has a number now, and the numbers start at zero. That seemed too obvious to need a thought, and it deserves one.

An hour ago you sent Mira a file. Your program used the ephemeral port 51844 and hers was on 5222. The transfer finished, both programs closed their sockets, and the four-tuple was released. Now you send her another file. Your operating system needs an ephemeral port, and although it has thousands to choose from it cycles through them, and an hour is long enough to come round again, so it hands you 51844. That is the same four-tuple as before. Piece zero, piece one, piece two, and off they go.

Meanwhile, somewhere in the middle, a piece from the old transfer has been having a bad day. It got stuck in a congested queue, then was rerouted around an outage, then stuck again. It has been in flight for the better part of an hour, which is absurd but not impossible. Its header gives the four-tuple 203.0.113.9:51844 → 203.0.113.7:5222 and the piece number 7. It arrives at Mira’s machine just as the new transfer reaches piece 6.

Her operating system looks at the four-tuple, and it matches the current conversation. It looks at the number, and 7 is what it expects next. It slots the piece into position 7 and acknowledges it. The real piece 7 arrives a moment later and is discarded as a duplicate. Mira’s program reads a file with fourteen hundred and sixty bytes of a different, hour-old file spliced into the middle of it.

Nothing detected this. The check from Chapter 3 passed, because the piece was not damaged, only late. The numbering from Chapter 11 passed, because the number was right. Every agreement held. The piece came from a conversation that no longer existed, and because every conversation starts its numbers at zero, an old piece’s number will always fit a new one.

The fix is to make old numbers not fit. If each conversation begins its numbering at a different place, somewhere unpredictable, spread across a large range, then a piece from an old conversation carries a number nowhere near what the new conversation expects, and it is rejected as out of range. The ghost still arrives. It fits nowhere, and it is dropped. So you pick a starting number at random, from a space of about four billion, each time a conversation begins.

That creates a new problem at once. If the start is not zero, Mira does not know what it is, so you have to tell her, and she has to tell you hers, because she will be numbering pieces in the other direction. Before any data can flow, the two sides need to agree on two starting numbers. That agreement is the beginning of the conversation, and it is natural to ask how many messages it takes. Let’s try to do it in two.

You send a message saying that you would like to start a conversation and that your pieces begin at 4,000,000. Mira replies that she understands and that hers begin at 9,000,000. Now you know both numbers. You start sending data from 4,000,000 and you know to expect hers from 9,000,000, and it appears to be done.

Look at it from Mira’s side. She sent her reply. Whether it arrived she has no idea. If it was lost, you are still waiting for it, and she is about to start sending data numbered from 9,000,000 to someone who does not know that is where her numbers begin and who will discard every piece as out of range. She cannot safely start until she knows you have heard her, and after two messages she does not.

So there is a third message, in which you say that you have it and that hers begin at 9,000,000. Now Mira knows that you know. Each side has said its number and each side has heard the other confirm it. Three messages, and not one fewer. The second message does two jobs at once, confirming your number and proposing hers, but the last proposal always needs a confirmation of its own, and that is one more message.

This is the three-way handshake, and the messages are traditionally called SYN, SYN-ACK and ACK, for synchronise, synchronise-and-acknowledge, and acknowledge. It lands on Chapter 8 in a satisfying way. The first message is the knock, the thing that makes a listening socket ready. The third message is what completes the conversation, and it is at that point, not before, that the operating system creates the connected socket and queues it for the program to accept. The handshake is the physical reality behind accept.

The third message also carries a bonus. You do not have to send it empty. You can put your first piece of data in it, numbered 4,000,000, and it serves as both the confirmation and the beginning of the transfer. So a conversation costs one round trip to set up, there and back, and the data starts flowing on the second trip out. Remember that cost. In Part II it turns out to be most of the wait for anything small.

The handshake also gives the connection refused of Chapter 8 a precise meaning. Your first message arrives at Mira’s machine and there is no listening socket on 5222. Her operating system cannot just ignore it, because you would wait, your timer would fire, you would send it again and again, and you would sit there a long time before giving up. Instead it sends back a short, blunt message saying that there is no such conversation and you should stop. Your operating system receives it and reports the refusal to your program immediately. That blunt message is called a reset, and it is the operating system’s way of saying that whatever you think is happening here is not. You will see it again whenever one side has forgotten a conversation the other side still believes in.

Starting a conversation was symmetric, since both sides had to propose and confirm. Ending is not symmetric, and the reason is that the two directions of a conversation are independent. Suppose you have sent Mira the whole file and you are done sending. Mira is not done. She is going to send you back a receipt, or a checksum, or a thumbnail, once she has processed it. If closing the conversation meant closing both directions at once, you could not say that you were finished without also cutting off her reply.

So closing is done per direction. You send a message that says you have nothing more to send, which we can call a FIN, and Mira acknowledges it. Your direction is now closed and hers is still open. She can keep sending for as long as she likes, and you keep acknowledging, and when she is done she sends a FIN of her own, and you acknowledge that. Now both directions are closed and the conversation is over. The state in the middle, with one side finished and the other still sending, is called half-closed, and it is a perfectly legitimate place for a conversation to be. A program that uploads a file and waits for a result lives there for a while on every upload.

Now look at the very end. Mira sent her FIN and you sent the acknowledgement. From your point of view that was the last message; both directions are closed, you are done, and you can forget the four-tuple and reuse the port. Before you do, suppose that your last acknowledgement was lost. Mira never receives it. Her timer fires and she sends her FIN again. It arrives at your machine, and if you have already forgotten the conversation, your operating system has no idea what this is and sends a reset. Mira’s program, which was closing cleanly, gets an error instead of a clean close. The last message of a conversation is the one message that cannot be acknowledged in turn, so the side that sends it can never be sure it arrived. The only remedy is to stay around long enough that, if the other side resends its FIN, you are still there to acknowledge it again.

There is also the ghost from the top of the chapter to think about. Pieces from this conversation might still be in flight, slow, rerouted, stuck. If you reuse the four-tuple immediately and one of them turns up, you are back to splicing hour-old data into a new transfer. Random starting numbers make that unlikely, and not reusing the four-tuple until the stragglers have died makes it impossible. So the side that sends the final acknowledgement does not forget the conversation. It holds the four-tuple in a waiting state, with no data and no program attached, just a placeholder, for long enough that any straggler would have expired and any resent FIN would have arrived. Then it lets go.

This waiting state is called TIME_WAIT, and it is another instance of the question you met last chapter, of how long to wait. The answer, again, is a guess with a margin, a couple of minutes on most systems, chosen to be comfortably longer than any piece could plausibly survive in flight. It is a timer that exists purely so that the end of a conversation cannot corrupt the beginning of the next one.

It has a cost that you will feel later. A machine that opens and closes many short conversations to the same peer, thousands a second, say, accumulates thousands of four-tuples sitting in TIME_WAIT, each one holding an ephemeral port that cannot be reused yet. Open them faster than they expire and you run out of ports entirely, and new conversations fail for a reason that has nothing to do with anything being broken. When that bites you, it will be because of this paragraph.

The numbered stream from Chapter 11 has become a connection: a thing with a beginning that both sides agree on, two directions that close independently, and an end that is careful about ghosts. That is the whole of what people mean when they say two machines are connected. There is no wire and no circuit, only a pair of operating systems that have agreed on numbers and are keeping track.

The handshake makes one question sharp. Every step in a connection’s life so far has been announced, with a message to start and a message to close each direction. It remains to ask what happens if there is no message, if Mira’s machine stops without saying so.