Too Big for One Frame

Mira sends you a photograph. It is a picture of the courtyard from her window, and it comes to about three megabytes, three million bytes give or take. Her chat program hands it to the socket in the ordinary way, and her operating system sets about building frames out of it. It gets no further than the first one, because the photograph will not go into a frame at all.

The trouble is the frame itself. Look back at the frame you settled on in Chapter 2. The length field was two bytes wide, which allows a content length up to 65,535 bytes. That is already far too small for a photograph, but the real ceiling is lower than the field allows. On the wire you have been building on, the content of a frame is capped at about fifteen hundred bytes, and the reasons for that are practical rather than fundamental. There are three of them.

The first comes from Chapter 3. When a frame arrives damaged, the whole frame is discarded, so a bigger frame means more bytes thrown away for every flipped bit. On a wire that flips one bit in a hundred million, a three-megabyte frame is twenty-four million bits, and one photograph in four would arrive damaged and be discarded entire. Small frames make errors cheap.

The second comes from Chapter 4. On a shared wire, a frame occupies the whole wire for as long as it is being sent, and a three-megabyte frame would lock everyone else out for a noticeable fraction of a second. Switches removed the sharing, but the habit stayed, and so did the equipment that had been built around it.

The third is memory. Every card and every switch has a fixed amount of it set aside to hold a frame while it decides what to do with it. That memory was sized for frames of about this length, decades ago, by a great many manufacturers, and changing it now would mean changing everything at once. So there is a ceiling, and it is not something you derive. It is handed to you as a property of the wire, in the way its speed is. The field calls it the maximum transmission unit, or MTU. A frame’s content can be at most about fifteen hundred bytes. A photograph is three million.

There is only one thing to be done about this, and you already know what it is. You cut the photograph into pieces that fit. Three million bytes, at about fourteen hundred and sixty bytes per piece once the headers have had their share, comes to something like two thousand and fifty pieces. Each piece is wrapped in a frame of its own, with its own location header and its own port header, and sent. Mira’s operating system receives the pieces, strips the headers off each one, and glues the contents back together in the order they arrive. Her program reads three million bytes from the socket, and there is the courtyard.

On a quiet wire, in one building, through one switch, this works without a flaw. The pieces go out in order, they arrive in order, they are glued in order, and the photograph is exact to the byte. You could stop here and be pleased with yourself, and for a while you are. Then Mira moves to a flat two routers away and sends you another one.

Between your machine and hers there are now boxes. They are routers, and they have queues in them, and there may be more than one path through them. The axiom, if you go back and read it, said that the wire carries bits in order. It said nothing about a path. It said nothing about what happens when your frames leave one wire, sit in a box for a moment, and are put onto another. Three things happen, and they happen together, on the same transfer. Each leaves a different mark on the photograph, and each is going to need a different cure, so we will take them one at a time.

The first thing that happens is that pieces arrive out of order. A router in the middle has two ways to reach Mira’s building and is spreading traffic across both. Piece 1042 goes left, piece 1043 goes right, and the right-hand path happens to be a little shorter today, so piece 1043 arrives first. Or there is only one path, but the router held piece 1042 in a queue for a fraction of a millisecond longer than 1043, because of how its internals happened to schedule them, and the two came out swapped.

Mira’s operating system, gluing pieces together in the order they arrive, puts the content of 1043 before the content of 1042. In the photograph this is a horizontal band, a strip of the courtyard a few rows high, exchanged with the strip below it. The picture is almost right. A brick wall has a seam in it that is not there in the real wall. Every byte arrived and every byte was correct. They were glued in the wrong order, because arrival order and sending order stopped being the same thing the moment there was a box in the middle.

The second thing that happens is that a piece goes missing. Somewhere along the path a router’s queue was full when piece 1500 arrived. The queue is a fixed amount of memory, it was holding as many frames as it could, one more came in, and the router did the only thing it could and dropped it. There was no error and no notice. Piece 1500 does not exist any more.

You have met this before, in a way. Chapter 3 told Mira’s card to discard a damaged frame, silently, because there was nothing else honest to do. That was the tower choosing to lose a frame. This is a box in the middle making the same choice for a different reason. Either way, the tower already loses frames on purpose, and loss is not an accident that befalls it from outside. It is built in.

What loss does to the photograph is worse than the swap. Mira’s operating system glues pieces 1499 and 1501 together as if 1500 had never been sent, and from that point on every byte sits fourteen hundred and sixty positions too early. The image decoder reads rows of pixels, and every row after that point begins in the wrong place, so the bottom half of the photograph shears sideways into diagonal streaks of colour. The total is also fourteen hundred and sixty bytes short, so the decoder either complains or runs out of data before it reaches the last row. One missing piece has destroyed half a photograph.

The third thing that happens is that a piece arrives twice, and this one needs a word of honesty. At present the tower has no mechanism that sends anything twice, so a duplicate can only come from the middle. A box, uncertain whether a piece got through, may forward it again; or a path may briefly loop, so that a frame goes round and arrives a second time. Both do happen, but they are rarer than the other two failures.

The reason it matters anyway is that you are about to fix loss, in the next chapter, and the fix will involve sending pieces again when you believe they did not arrive. Sometimes you will be wrong. The piece did arrive, and what got lost was the evidence that it had. So you send it again, and now Mira has two. Every honest cure for loss creates duplicates, and if you do not plan for them now the cure will bring them back with interest.

What a duplicate does to the photograph is a missing piece run backwards. Piece 1500 arrives, and then arrives again, and Mira’s operating system glues both in. From that point every byte sits fourteen hundred and sixty positions too late. The bottom half shears the other way, and there are fourteen hundred and sixty bytes too many at the end.

The three signatures are easiest to compare when they sit next to each other, so put them side by side. The whole of the next chapter depends on telling them apart.

sent:     … 1041 1042 1043 1044 … 1499 1500 1501 …

reorder:  … 1041 1043 1042 1044 …                   a band swapped
loss:     …             … 1499 ──── 1501 …          everything after shifts early
duplicate:…             … 1499 1500 1500 1501 …     everything after shifts late

In reordering, every piece is present and the sequence is wrong. In loss, a piece is absent. In duplication, a piece is present twice. The photograph comes out wrong in three different ways, and the receiving operating system cannot at present tell that any of them happened. Pieces arrive and it glues them. It has no idea what order they were supposed to be in, how many there were supposed to be, or whether it has seen this one before, and it does not know because you never told it. Nothing in any header says that this is piece 1042 of 2050.

One thing should be pinned down before we go on, because it is easy to feel cheated by it. The axiom said the wire carries bits in order. That was true and it is still true. On any single wire, between one card and the next, frames come out in the sequence they went in. What you have discovered is that a path is not a wire. A path is a chain of wires with boxes between them, and every box is a place where frames wait, and waiting frames can be released in a different order from the one they arrived in, or dropped, or occasionally sent on twice. The axiom held. It held for a smaller thing than you had assumed.

This will matter a great deal in Part II, when we look closely at what the boxes do promise. For now it is enough to say that order was a gift of the single wire, and you have left the single wire behind.

So you have a broken photograph, broken in three ways on the same transfer, by things the tower does on purpose or cannot prevent, and the operating system that is gluing it back together does not know, because the pieces carry nothing that would let it know. You cut the photograph into pieces because there was no other way to get it onto the wire, and that was necessary; it is the rung. The pieces are now at the mercy of the path. In the next chapter you give each piece the one thing it is missing, which is a number.