The Shape of the Message

Courtyard is working. Chat frames carry messages, the feed serves photos, and the shape of every message is settled. Then Mira improves it.

Inside every chat frame from Chapter 18, after the type and the length, is the body, and the body has a shape of its own. A text message, say, is three things: who sent it, what they said, and when. You laid it out the obvious way, in binary, because the frame was binary.

┌──────────────┬────────────────┬───────────────────┬──────────────┐
│ sender id    │ text length    │ text              │ timestamp    │
│ 4 bytes      │ 2 bytes        │ as many as length │ 8 bytes      │
└──────────────┴────────────────┴───────────────────┴──────────────┘

The reader knows the layout. It reads four bytes, and that is the sender; two bytes, and that is how much text follows; that much text; eight bytes, and that is the time. It is fast, compact and unambiguous. Everyone running Courtyard, which by now means you, Mira, Theo, Priya and a few dozen others, has a client that reads this layout and no other.

Mira wants replies. When you reply to a specific message, the client should show which one, so a text message needs a fourth thing, the id of the message being replied to. She adds it in the sensible place, right after the sender.

┌──────────────┬──────────────┬────────────────┬──────────┬──────────────┐
│ sender id    │ reply to     │ text length    │ text     │ timestamp    │
│ 4 bytes      │ 4 bytes      │ 2 bytes        │ ...      │ 8 bytes      │
└──────────────┴──────────────┴────────────────┴──────────┴──────────────┘

She updates the server and her own client, and sends a message. Your client, the old one, reads four bytes and takes them for the sender. Then it reads two bytes for the text length. But those two bytes are now the first half of the reply-to field, which for a message that is not a reply is zero, so the text length is zero and it reads zero bytes of text. Then it reads eight bytes of timestamp, which are actually the second half of the reply-to field, the real text length, and the first four characters of Mira’s message. The timestamp is garbage and the text is empty. Your client shows a blank message from Mira dated somewhere in the year 47,000.

Nothing crashed, nothing reported an error, and every read succeeded. This is Chapter 2’s failure, plausible garbage delivered silently, brought about not by a shifted bit stream but by a shifted layout. It is the same shape of problem, six floors up.

Mira did nothing unreasonable. She added a field to a message. What she could not do was update every client at the same instant, because she cannot reach into your machine and upgrade your program. So for a while an old reader is looking at a new message. That “for a while” is the whole subject of this chapter, because it never ends. There is always an old reader somewhere.

Suppose, then, that she appends instead of inserting, and puts the reply-to field at the end, after the timestamp. Now an old client reads sender, length, text and timestamp as before, and finds four extra bytes left over, and the question is what it does with them. If it was written to read the fields it knows and stop, it is fine, and the four bytes are ignored. If it was written to check that it consumed the whole body, it rejects the message as malformed. If it treats what follows the timestamp as the start of the next message, which is plausible if the code was careless, it is back to garbage.

So appending can work, but only if the reader was built in advance to tolerate bytes it does not understand. That is a rule about the reader, and it had to be in place before the change happened. It is the first rule, and we will collect the others as we go.

The timestamp, it turns out, is better set by the server than by the client, so Mira stops sending it. The new message is sender, length and text, three fields. Your old client reads sender, length and text, and then tries to read eight bytes of timestamp from a body that has ended. Depending on how carefully it was written, it either crashes, or reads eight bytes of whatever comes next in memory, or reads the first eight bytes of the following frame and desynchronises the whole stream. None of these is a blank message dated 47,000. All of them are worse. Removing a field an old reader expects is never safe. No rule on the reader’s side can save it, because the reader is looking for something that is not there.

Every problem so far comes from the reader knowing fields by position. Field two is the length because it comes second. Insert something and every position after it shifts; remove something and the reader runs off the end. So give the fields names instead. Label each one, and have the reader find fields by label rather than by offset.

{ "sender": 7, "text": "meet at 7", "timestamp": 1726480801 }

Now Mira can add "reply_to": 412 anywhere in that object, and your old client, looking for sender, text and timestamp by name, finds them regardless of where the new one sits. The first rule, that a reader ignores what it does not recognise, becomes natural. A reader that looks things up by name does not look up the names it does not know.

That is a real improvement, and it is why almost every message format that has to evolve uses names or tags rather than bare positions. But it does not make you safe, as the other changes show. Suppose Mira removes timestamp. Your old client looks it up by name and it is not there. If the client assumed it would be there, and it did, because it always was, the client fails in whatever way looking up a missing thing fails in its language. Naming did not help. The field is still gone.

Suppose instead she renames text to body. Her new server sends body. Your old client looks for text, finds nothing, and shows an empty message. Meanwhile your client, when it sends, still sends text, and Mira’s new server looks for body, finds nothing, and stores an empty message. It is broken in both directions at once. A rename is a removal and an addition, and the removal half is fatal.

Suppose, finally, that she keeps the name and changes the meaning. She decides timestamps should be in milliseconds rather than seconds. It is the same field, with the same name and the same type, and values a thousand times larger. Your old client reads it perfectly and displays every message as arriving in the year 56,000. I find this the worst of the three. Nothing is missing, nothing is malformed, every parser on both sides succeeds, and every value is wrong. No check you could build into the format would detect it, because the format is fine. Only the agreement about what the number means has changed, and that agreement lives in people’s heads.

loading…

The shape of a message is an agreement, like the meaning of a lit lamp, like the table from letters to bits. It has the property every agreement in this book has had, which is that it works only if both sides have it. And it has a property none of the others had to face so directly: it changes over time, and the two sides do not change together. So compatibility has to be designed in both directions. A new reader must handle messages from an old writer that knows nothing of recent fields. An old reader must handle messages from a new writer that includes fields it has never heard of. Both hold, always, because you never control when everyone upgrades.

Five rules make both directions work, and you have met most of them already in the wreckage above. Readers ignore fields they do not recognise; this is what makes adding safe, and it has to be true of every reader before the first addition ever happens. Writers only ever add. A field that exists is a field some reader depends on, so if you stop needing it you stop populating it, mark it as obsolete, and leave it in place, because removing it breaks every reader that has not caught up, which is some reader, forever. Nothing is renamed, since a rename is a removal plus an addition; if the name is wrong, you add the right one alongside and populate both. A field’s meaning never changes, not its units, not its type, not what it refers to; if the meaning needs to change, that is a new field with a new name, and the old one keeps meaning what it always meant. And new fields are optional, with defaults, because an old writer will not send them and a new reader must be able to proceed without them.

Follow those five and any writer can talk to any reader, in any combination of versions, indefinitely. Break any one of them and there is a pairing somewhere that fails.

You might object that naming fields with strings, as above, is expensive, since every message now carries the word timestamp, and that this chapter has abandoned the compact binary layout Chapter 18 chose for chat. It has not. Consider what a named field actually needs to be. The reader has to be able to find a field it knows and skip one it does not, and to skip something you need to know how long it is. So each field has to announce its own identity and its own length before its content.

┌──────┬────────┬─────────┐ ┌──────┬────────┬─────────┐
│ tag  │ length │ value   │ │ tag  │ length │ value   │ ...
└──────┴────────┴─────────┘ └──────┴────────┴─────────┘

There is a tag, a small number standing in for the name, then a length, then the bytes. A reader walks the fields. For each one it reads the tag, and if it knows the tag it reads the value, and if it does not it reads the length and skips that many bytes. Unknown fields are ignored by construction. The layout is binary and compact. And the tag plays the role of the name, so the rules above apply with “tag” in place of “name”: never reuse a tag number for a different meaning, and never change what a tag’s value means.

loading…

Look at that field: kind, size, content. It is the chat frame from Chapter 18, and it is the frame from Chapter 2. The message body is made of frames, each of which is made of framed fields, and every level solved the same problem the same way. When a message format is described as self-describing or extensible, this is all that is meant. Every piece says what it is and how long it is, so that a reader who does not understand it can at least get past it.

There is an obvious tool this chapter has not reached for, which is to put a version number in the message. It is tempting. Version 3 messages have these fields in this layout, version 4 has those, and the reader checks the number and parses accordingly. Now consider what that costs. Every reader must understand every version that any writer might still send, and old writers exist for a long time. You can never retire the parsing of version 3 while a version 3 client is out there, and you have no way to know when the last one is gone. A new writer, meanwhile, has to decide which version to send, and if it sends the newest, every old reader that does not know it is broken. So writers end up sending old versions to be safe, and the new version is used by nobody. Version numbers do not solve compatibility. They give you a label for the incompatibility.

They do have a place. When a change is truly breaking, when the old shape cannot express the new meaning and additive evolution has run out, a version number lets both shapes coexist while the world migrates. That is rare and expensive, and it should feel that way. Day-to-day evolution, a field here and a field there, month after month, is done by the five rules, additively, with no version numbers at all. A format that has been following the rules for years has only ever had one version, and it has changed constantly.

One last consequence remains, and it is the one that will bite hardest later. Mira’s server does not only send messages; it stores them. Every chat message and every photo’s metadata sits in a file on her disk in whatever shape the server that wrote it used. She deploys a new server that writes the new shape. A week later there is a bug, and she rolls back to the old one.

The old server now reads the messages the new one wrote. This is an old reader and a new writer, the same problem as before, except that the writer was Mira’s own server a week ago and the old reader is her own server today. Data at rest is a message from the past to the future, and the past and the future are two versions of the same program that never run at the same time. Every rule above therefore applies to storage as it applies to the wire. A field removed from the storage format breaks every rollback and every record written before the removal.

You will meet this properly in Part III, when forty teams share the data and nobody remembers who added which field. For now the picture only needs extending. A message is not only a thing that crosses a wire. It is any bytes that one version of a program hands to another, across space or across time.

Courtyard can change. Mira can add replies, and later reactions, and later whatever she likes, and clients that have not updated keep working, showing what they understand and ignoring what they do not. The agreement about shape evolves without ever breaking. The next chapter takes up a question you have technically been able to ask since Chapter 18 but have not, which is what happens when a request goes out and no reply comes back.

Coming up · I·20 — Safe to Ask Again?

You tap send. The connection hiccups. No response. Do you send it again? If Mira’s server already got it, she will see your message twice, and everyone has seen that happen.

→