worm-blossom

worm-blossom

Mundane Numbers

A convention for magic numbers in binary (i.e., non-plaintext) file formats.

  1. Generate a random 128 bit integer.
  2. Set the five most significant bits to one.
  3. Each file of your file format must start with the big-endian encoding of that number.

Thus ends the specification, now starts the rambling.

Rationale

TLDR: 123 bits are sufficient to avoid random collisions, and the five most significant one bits immediately indicate that the file is not plaintext.


Magic numbers at the start of non-plaintext files have a long tradition. A typical magic number has up to eight bytes, and is cute (spells something out in ASCII, is the author's birthday, etc.). The “mundane numbers” I propose are not that. Why?

Let's start with the basics: long magic numbers consume more storage space (and bandwidth on transmission) than short magic numbers. The most efficient choice is to have no magic number at all.

Does advocating for 16 byte magic numbers make me a bad person then? For one, this depends on the expected size of files in your file format. If your file serialises database contents with millions of records, the magic number is effectively irrelevant. Second, even for tiny files, the overhead of file metadata is probably more relevant than your choice of magic number. And finally, in situations where performance is actually critical (many, many files, each itself quite small), the best move is probably to optimise the magic number away: store (or infer) that a directory contains a bunch of files of your file format, and then omit the magic number from those files.

Why magic numbers at all? Arguably to make it easy to determine how a file is supposed to be interpreted. Given that some files might just contain random binary data, magic numbers need to be sufficiently long to avoid accidental matches. Thankfully, 128 bit already suffice. 123 bit still suffice. 32 bit do not.

My main gripe with short, cute magic numbers is not the probability of matches with random files, however. It is the underlying assumption that the magic number you assign to your file format will not conflict with the magic number of anyone else's file format. Did you exhaustively check whether your magic number is already in use? Did you look up your number in the magical database of all magical numbers in existence at the current point in time? And do you expect all future file format definitions to somehow become aware of your magic number?

Files themselves are meaningless bytestrings, they can only be interpreted in a context. Magic numbers don't change that, they are only meaningful with respect to the set of magic numbers somebody is checking against. And that set varies across places, and across time. And the only way to ensure that set will not contain duplicates is to use sufficiently long, random numbers.

As software developers, we should leave behind the idea that it is easy (and will remain easy, and, for that matter, has ever been easy) to achieve global consensus on anything. Especially for something as uncoordinated as definitions of file formats. Anyone can define their own file format and start using it, and people can do this concurrently, without coordination. This is good, it should stay this way, and we should design for a world in which this is the case. And that means choosing robust magic numbers, not being parsimonious or overly cute. Whenever I see file formats with short, nonrandom magic numbers, I see a design culture that tacitly assumes centralisation and rejects pluralism.

To be clear, I have no qualms with completely foregoing magic number either; relying on external metadata is perfectly fine. Though, in brief, my arguments in favour of having a magic number are: