worm-blossom

A New Era Dawns

For the past year, we have been pretty busy here at worm-blossom. We have also been pretty quiet. But that is about to change. From now on, you can expect weekly updates about our progress on this very website. As you can probably tell already, we will restrict our reports to the minimal technical details, the bare necessities, the austere metrics of productivity. Nothing else.

Throughout our work on Willow, we increasingly had to take on technical debt in order to meet deadlines. Combined with our inclination of keeping work to ourselves until it is just perfect, we ended up accumulating a vast body of work we stayed fairly quiet about. Not only did this make us look stagnant, it also made us unhappy. But now, with our previous grant obligations fulfilled, and the next grant hopefully leaving us more breathing room, we decided to take action. Going forward, we will

• identify minimally useful subsets of everything we do and publish these as early as possible,
• apply this policy not only to future work, but also to our backlog of pretty cool accumulated things,
• emphasise the exuberance and creative spark that drives us to do our work,
• invite and mentor contributions to our projects, and
• share our progress in the most joyful way we can on this website.

And what better way of holding us accountable to these lofty resolutions than to boldly and pinkly announce them on the world wide web?

~Aljoscha

This week we renamed two Willow specs: Willow General Purpose Sync and the Sideloading Protocol. I think it's important to sit up and notice when every time someone hears the name of your project, they say "like GPS??? Huh???". And at this point the term 'sideloading' has such a toxic association with locked down mobile platforms that it felt like a completely unforced error.

We renamed them to Willow Confidential Sync and Willow Drop Format, respectively. Not only are these nicer names, but it also meant I could redo the emblem for the WGPS.

The WGPS was the only spec which didn't have a nature-inspired emblem, and this has now been remedied. I also remedied the total absence of pink from the lineup. It's a grapevine, because with confidential sync you "heard it through the grapevine". Isn't that precious?

~sammy

This week I put a lot of time in non-willow projects. The biggest timesink was Ufotofu, which is a library for working with lazy sequences (“sinks” and “streams”) in Rust. Specifically, I wrote tons of documentation, tests, and documentation tests in preparation for a proper release. This crate has served us well for our async needs while implementing Willow in Rust, and I am giddy about getting to share it more widely soon!

I also put some time into Macromania, our typescript-embedded templating language we use to generate willowprotocol.org in all its hyperlinked and tooltippy glory. We have accumulated a couple of breaking changes we wanted to do to the core of Macromania, and those are now all implemented. Sadly, this means that all the fancy macros powering tooltips etc do not work with that new version of Macromania.

As for the current state of Macromania: the core library is now stable and has pretty solid doc comments (the curious can look at the jsr release), but we still lack high-level documentation for onboarding folks into the madness. We will start working on a proper website featuring this high-level documentation soon-ish (for me, ufotofu has higher priority at the moment). That website might turn out fairly unimpressive initially, because we still lack all the really powerful macros — not only did we break their foundation, but we have also accumulated a bunch of breaking improvements we want to roll out for most of our core macros. Nevertheless, the new version of Macromania already builds this website, and does some fun stuff such as generating an RSS feed and highlighting code snippets. See the source and tinker around if you are feeling adventurous!

This was a highly productive week from my side for ufotofu and Macromania, yet I feel twinges of guilt. Both of these projects are fairly computer-centric, not human-centric. Well, ufotofu in particular; it tries to apply some mathematical principles to streamline a programming experience. Programming for programming’s sake, as if the world could seriously be improved by making computers and programming more efficient and thus even more entrenched... But still, I cannot deny that this kind of work is extremely fun and satisfying to me. In weeks like this, I carry with me some cognitive dissonance centered around escapism and indulgence, counteracted by telling myself that all of this keeps me motivated and will ultimately further Willow (not only by keeping me engaged, but also because our Willow implementations rely on ufotofu and our web presence relies on Macromania).

I want to end on a more light-hearted note, so, erm, have some borderline hyperactive sketches for two pianos (loosely inspired by listening to some Meredith Monk pieces):

Also, *how* is “instalment” correct spelling? ~_~

~Aljoscha

This week I have been mostly single parenting, so I did not get much programming done. Most of my creative energies went towards cooking meals.

For breakfast I often turn to the humble porridge. I like to put big chunks of apple in my porridge which cook just long enough to get soft but still keep their structure, kind of like in an apple pie. I like to cook my porridge low and slow, which really annoys everyone else.

You can't go far wrong with a curry. I break up a cauliflower into little florets and roast them in the oven until they start to caramelise at the edges. I could eat them just like that, but in a curry sauce they're even better. I make a curry sauce from scratch and it is completely inauthentic.

Salmon is great when you're low on time because you just smack it in the oven for a bit. In the colder months I like to do a roast vegetable mix alongside it: potato, beetroot, sweet potato, red onion, topped with some capers. The purple of the beetroot and onions look so good next to everything else.

And when it all became too much for me I just ordered pizza. hashtag winning

~sammy

The Big Giant Willow Website Update

willowprotocol.org, the online home of our peer-to-peer storage protocols, has been updated to reflect all of the work that we’ve done on Willowfrom April 2024 until now (work funded by our friends at NLnet! Hartelijk dank!). That’s eighteen months of work in a single serving! Argh!!! So what are the highlights?

Firstly, we’ve reworked Willow Confidential Sync(neé W.G.P.S) to be man-in-the-middle attack resistant. In the old protocol, if two peers who did not know each other before were to sync over a compromised network (e.g. a network with a router modified to intercept traffic), an attacker would have been able to access all sorts of Willow-stored data! As Confidential Sync is expressly designed for secure syncing between untrusted peers, we've redesigned Private Interest Overlap to mitigate this kind of threat. You can learn more about the security properties and threat model of this design here.

The Willow Drop Format (neé Sideloading protocol), which encodes a user-selected subset of Willow data into a single binary (ready to put on a USB key or send via Signal), now is able to add entries without their payload. We’ve also made the encoding more efficient so you can squeeze even more Willow data into a smaller number of bytes.

We’ve added a new page for Willow’25, a recommended set of efficient, secure parameters for all the Willow specs. This way, nobody needs to witness the nightmare of generic types we’ve created for ourselves unless that’s what they really want.

There’s a new Rust section which has tutorials for the crates we’ve released so far. Not only does this section have a completely gratuitous illustration of Willowverse mascots Alfie and Betty riding a giant crab, but we’ve also integrated rustdoc support into willowprotocol.org’s referencing system. This means that our tutorials can reference things like structs and their methods, and link readers directly to up-to-date Rust API documentation. We will be regularly updating this section over the coming weeks and months!

There are new illustrations, diagrams, and comics pretty much everywhere. The Drop Format spec. The Private Interest Overlap Page. Encrypting Entries. The ‘More’ section. The About us page.

Another page which has lots of new illustrations is the front page, which we completely redesigned to make answering the question of “what is Willow?” a bit easier. It also has a big fun illustration for desktop users.

All of this work got stuck in a kind of mega branch comprised of 67,020 additions and 14,590 deletions over 330 commits with the sole description of ‘do not merge yet, wip’. No comment.

~sammy

There might exist gifted programmers who can write non-trivial code and get things right the first time. I am not among them. But still, there is no way around implementing everything for a first time.

Over the past year, we have implemented most everything we specify on the Willow website in Rust. For the first time. Technically, that code does what it is supposed to do. But a lot of it is cumbersome to use, and there are several structural problems I really don’t want to get into (here). Long story short: we are reworking that code, and re-releasing it over the coming weeks. And from what we can tell so far, the new code is pretty enjoyable to work with!

We started this week by releasing an implementation of the Willow data model. Here is the plan for what will follow:

• Groupings (a requirement for any non-trivial Willow usage),
• Meadowcap (at that point, you can start sketching out access-control for Willow-based applications),
• traits for (persistent) storage for entries and payloads (at that point, you can write full application logic against the traits),
• simple in-memory and persistent implementations of the store traits (at that point, you can write and run full offline-only applications), and
• the Willow Drop Format (at that point, you can write and run full sneakernet applications).

Once these are done, we will probably slow down a bit, and play around with our own application ideas, end-to-end encryption, network-based synchronisation, and anything else that might come up. In addition to working on Bab, Ufotofu, and Macromania, that is.

~Aljoscha

I accidentally published an incredibly nerdy but surprisingly useful Rust crate this week: order_theory.

The Willow specs often refer to minimal and maximal values (such as SubspaceIds or Paths). Sometimes we also need a notion of successors on these types. So I defined some traits capturing these concepts. And then I moved them into their own crate, because I needed them in several places. And since you cannot very well publish a trait claiming to provide order-theoretic concepts without traits for lattices, I added a corresponding hierarchy of traits as well.

Then I needed our NamespaceId and SubspaceId types to implement SuccessorExceptForGreatest. So I added a generic implementation of all order-related concepts of our new crate for arbitrary arrays, and simply delegated to those from willow25.

Next, I wanted to convert arbitrary Rust ranges (which feature both inclusive and exclusive bounds) to Willow ranges (which are less flexible). Turns out that you can do those conversions exactly if the type in question is totally ordered and has a notion of predecessors and successors. Good thing I had recently defined traits for exactly that!

All Willow groupings support a notion of intersections — the intersection of two groupings is the grouping which contains exactly the entries contained in both original groupings. Why, that sure sounds like a LowerSemilattice. How convenient I had a trait for that concept.

Finally, implementing intersections (and other order-related methods) is not always easy. So I added utilities for property testing to the order_theory crate. And these then found several errors in our previous Willow groupings implementations.

Math! Yay!

~Aljoscha

Not much to report this week as I was travelling. The coming week is going to be busy as November is here, and 2026 is starting to rear its fuzzy little head. I think it's going to be a good year, but only if I write all the proposals and applications like I'm supposed to.

~sammy

worm-blossom: now on Fridays. We have moved the publishing day because having a Monday deadline put a lot of pressure on the weekend. On Thursdays I'd think I have so much time to write about the French revolution, handily forgetting that I never have the opportunity to use a computer on the weekends. Then Monday would turn into a very stressful day. Now Fridays can be a very stressful day instead, which is much better.

We sadly had our proposal for Rummager rejected this week. It's one of those ideas that is very hard to convey, and I'm starting to think we'll just need to build out a prototype of it in our spare time to be able to do that.

It's a bit similar to something else I noticed this week: no matter how much documentation we write — loaded with illustrations and references and sidenotes — you just can't build up the same kind of understanding as if you actually got your hands on it. I'm thinking of building out some interactive tools for the Willow Data Model page because of that.

In any case, we have no shortage of (funded) plans. Keep muddling through out there.

~sammy

The codec traits we have added to Ufotofu today have been part of older ufotofu versions for quite a while now. And they have been exceptionally useful throughout our Willow-in-Rust journey. In fact, they alone already justified ufotofu as a whole for us. What makes them so special?

Not just any value that can be turned into a bytestring gets to implement our Encodable trait. No, there are precise formal requirements that must be fulfilled by all possible encodings. Every possible value must have at least one encoding. No two nonequal values must have the same encoding. No valid encoding must be a prefix of any other encoding. That sort of thing (see the docs for the precise list). But still, how is that supposed to be incredibly productivity-boosting?

The answer lies in the humble codec::proptest module. It provides functions such as assert_codec. You call that function with a bunch of values, and the function panics if and only if the arguments are a counterexample to any of the formal criteria an encoding must fulfil.

We use this to let a coverage-guided fuzzer try to find counterexamples. And if it cannot, then we can be pretty darn confident that our implementation is correct. Which only few of our initial implementation attempts are. But we just press a button, and the computer gives us a set of failing inputs. No matter how many encodings we implement (and Willow alone has quite a few), we always get an almost ideal test setup and debugging aid for free.

And even better, this process has uncovered plenty of faulty definitions of encodings in the Willow specs as well. Some of our encodings are quite fiddly, and it is easy to lose track of some detail and specify an encoding which cannot be uniquely decoded, for example. But if the fuzzer cannot find a counterexample after 20 minutes, we can be highly confident that the encoding we specified is free of bugs.

Now that we have properly released ufotofu and its tooling around encodings, we will return to the Rust implementation of Willow proper. If everything goes according to plan, we will release a cleaned-up, well-documented, and ergonomic-to-use implementation of Meadowcap next week. And internally, it will be powered by ufos and tofu!

~Aljoscha

This week I made good on my threat to build an Interactive Willow Data Model. It's a toy version of Willow which lets you view and add entries and 'sync' between three 'peers'.

Is having something to play with going to be the missing piece of our documentation story? I don't know. It's made me want to write documentation which intersperses theory with lots and lots of mini-toys, but who has the time?

Most of the work on this was sunk into two spots:

• Making it so you could get an instant preview of what your changes would look like before committing them. This was something Aljoscha considered a must-have after looking at the first version, and unfortunately I had to agree with him.
• Squeezing a lot of (variably sized) information into a snug three column layout. Lots of overflow to deal with.

The most unsettling part of working on this by far was how powerful I felt working with TypeScript and (P)react again. After a year plus of working in Rust, I thought I'd be wringing my hands at the sheer barbarity of it all, but instead I was completely at home in the filth.

~sammy

On Variable-Length Integer Encodings

Most competent programmers are aware that linked lists are inefficient data structures, simply because they do things which modern hardware struggles with. Having indiviual elements point out to the next element (if any) is much less efficient than simply grouping elements next to each other in memory and prefixing them with the number of elements. The opening paragraph of the wikipedia article on linked lists points this out. The documentation for linked lists in the Rust standard library points it out in CAPS.

Here is a hot take: LEB128 — the most widely-used variable length integer encoding — is the linked list of varints. The basic idea is to represent an integer as a sequence of bytes. The first seven bits of each byte encode information about the integer, and the final bit indicates whether the end of the encoding has been reached or not.

For computers, this format is a pain to decode. Modern hardware wants to operate on multiple bytes simultaneously whenever possible (even without SIMD, processing still happens at words of multiple bytes), but an LEB int must be processed byte by byte. And for every single byte there is a branch which is highly difficult to predict.

Analogous to data structures for sequences, a much better design is to provide the length up front and to then have the actual data in a contiguous chunk. CBOR got this right, I humbly offer the VarU64 spec and the way we encode integers in Willow (which is essentially a generalisation of the CBOR design), other people have proposed similar designs. The exact choice between any of these barely matters. But they are all significantly more efficient than LEB128. And as a bonus feature, comparing their encodings lexicographically coincides with numerically comparing the encoded integers, which can be helpful when working with kv-stores — but is not the case with LEB128.

For the most part, this does not really matter. Very few systems will critically suffer from the performance penalty implied by LEB128. I just find it curious that so many competent engineers simply accept the very first design they stumble upon. And where is the warning in the first paragraph of the Wikipedia page on LEB128? Why does that page only provide links to “related encodings” that replicate the same design choice, instead of discussing alternatives?

Oops, looks like this week turned into a week of rants for me. Oh well, going forward I will at least be able to link to these pieces instead of repeating the same rants over and over in different discussions.

~Aljoscha

Equality is a notion we often use without paying close attention to it, and sometimes that can burn us. Take for example the notion of ranges in Willow. We implicitly assume a syntactic notion of equality: two ranges are equal when they have equal start and end points. An alternate definition might be a semantic one, where two ranges would be considered equal if they describe the same set of values.

If you allow ranges to be empty (i.e., ranges whose end value is less than or equal to their start value), you get a mismatch between the two definitions: the ranges [2..2) and [3..3) (and [7..4), for that matter) contain the same sets of values (none at all), but they differ syntactically.

Up until Thursday this week, we allowed empty ranges. After days of painful debugging during the reworking of our Meadowcap implementation, we do not anymore.

The trouble started innocently, while I happily ported our implementation of the EncodeAreaInArea encoding. This encoding works with an inner range of timestamps which must be included in some outer range of timestamps — i.e., the inner range must describe a subset of the timestamps described by the outer range. The encoding encodes the start time of the included range by adding some positive number to the start of the outer range. This works great for non-empty ranges, but not when trying to encode the empty range [3..3) inside the outer range [5..8). Oops.

A simple and efficient fix would be to normalise empty ranges when encoding: when encoding an empty range within the range [start..end), you could always encode it as [start..start). The problem: we have strict notions of what constitutes a valid scheme for encoding values in Willow. In particular, equal values must yield equal encodings, and nonequal values must yield nonequal encodings. Normalising empty ranges during encoding would violate those basic (and sensible) axioms. We have mixed syntactic and semantic notions of how to compare ranges in the same encoding, and the result was pain.

In our old implementation of ranges in Rust we forbade empty ranges, which is why this issue never materialised in actual test failures. Frustratingly enough, most of the time spent in updating our implementation of all entry groupings was in making sure that empty groupings would be handled correctly. And now we had to flat-out remove all empty groupings from the spec, which meant I had to refactor all that code again, mostly undoing the finicky work I did two weeks prior. Diagnosing what was going on and then refactoring all groupings took me several (unpleasant) days. And that is why the revisited Meadowcap took so much longer than I expected.

I hope I learned my lesson and will pay closer attention to (implicit) notions of equality in the future. I was partially aware of this issue already; we have always defined only nonempty intersections of areas, for example. But for ranges I somehow hadn’t spotted the issue. On the bright side, our principled definitions of the semantics of both groupings and of encoding functions meant that we did catch this issue after all — in fact, it resulted in actual failure of a property test. This is exactly the kind of specification problem that has the potential to cause a whole lot of pain years down the road. So as much as the delayed Meadowcap release annoys me, I am glad we now fixed the issue.

~Aljoscha

Two weeks ago, @Soapdog asked me whether I could put the weekly songs on this website into a playlist. And thus, a slow descent into madness began. The tool in which I write these tracks has absolutely no support for that kind of thing — but it is open source and can be forked. And thus, I proudly present the official worm-blossom playlists page. Featuring not only a playlist of all weekly songs (automatically populated by Macromania, obviously), but also a bonus playlist of a bunch of tracks I have accumulated in a plaintext file over the years.

The original tool for creating this music is Beepbox, but I have been using a “mod” of it, which really just means a fork. Or rather, a fork of a fork [...] of a fork. There’s a family tree of how these mods have evolved, and I decided to fork pretty close to the leaves, at the 2024 mod AbyssBox (primarily because it has a ring modulation effect). The codebase is a succession of mostly independent, mostly undocumented spaghetti modifications to an undercommented core. So everything took longer than expected, but eventually I succeeded in adding my own underdocumented spaghetti modifications!

The codebase I started from had no notion of multi-song containers at all. The first thing I had to add was code that would allow the synthesizer to emit a notification once it had finished playing its song. Next, since many of the songs are infinite loops, I added support for slowly fading out a track after some number of repetitions (fun fact: you need a logarithmic decrease in volume, not a linear one). Finally, at the start and end of playback there would always be an audible click sound, so I added a miniscule exponential ramp-in and ramp-out. I cannot unhear that click in other beepbox mods now!

As a bonus, since we have full control over the audio player now instead of merely embedding it, I adjusted the colour scheme to be even more worm-blossomy — and I could finally remove some vertical spacing that had been bothering Sam! All of this was a fun side-quest, but now I am completely done with it and will almost certainly not add any additional features such as a playlist-to-zip-of-mp3s export, earwig compatibility, or even custom sound synthesis to it.

~Aljoscha

For this week's comic, I composed an image to look like console output, printed it out, cut it out and glued it to the rest of the comic before scanning it back in and putting it on the internet. I hope somebody out there appreciates that. I make the original Rust Unwrapped image available here.

There's only one month left to do all the things I really wanted to do in 2025. Which means this week I got curtain bangs and started prototyping a video game. Will it be any good? If you never hear about it again, you will know the answer (the bangs look great though).

~sammy

Implementing Meadowcap over the past weeks has been quite draining, so I decided to take things more slowly this week. And then I didn’t — oops! We want to make progress toward a usable Willow implementation as quickly as possible, and the next step there is persistent storage. Which, in particular, means payload storage, which in turn means working on Bab.

We want to support pluggable storage backends with our Bab implementation: in-memory for testing, filesystem-backed for portable persistent storage, kv-store-based storage for an easier time getting transactionality working in a kv-based Willow store, etc. So what should the common interface of all these backends be? I wanted to keep this interface as simple as possible, and push as much Bab-specific logic as possible into higher-level components that can be shared between all implementations.

What I ended up with is a ByteStorage trait that is essentially a highly simplified, asynchronous Vec<u8>. You specify a total capacity up-front, you can read and write bytes within that capacity, and you can increase the capacity (if absolutely needed). That is all the funcitonality you will need to provide for a custom Bab backend.

The gap to Bab is bridged by treating such an array of bytes as an implicit representation of the Bab Merkle tree. Representing trees as flat arrays is a pretty standard maneuver, though we get some extra “fun” by allowing to specify that the bytes only store Bab chunk data starting at a certain offset (and up to a certain length, but that part is basically free to implement) — the whole point of Bab is partial verifiability, so it would be quite pointless if the underlying storage would have to store complete strings. As a result, I wrote a whole bunch of non-trivial recursive functions for various kinds of index computations (here’s an example). I’ve included a scan of the notes I took on the first day of work below.

Next, I found a neat way of implementing all the various optimisations for verifiable streaming (all centering around omitting as much data as possible while preserving the ability to verify things within a parameterisable, bounded amount of memory): the SliceProgressIterator takes the desired optimisations as an option, and sequentially emits all the Merkle tree nodes that are part of the verifiable stream, annotated with information on which data can be skipped. The nice thing about this approach is that it is fully decoupled from any particular storage backend, and can even be used to guide both producing and consuming a verifiable slice stream.

The final bit of work was to use this iterator in a function that actually produces a verifiable slice stream by reading data from a storage backend. Thanks to those earlier abstractions and helpers, that function looks quite tame, despite being a surprisingly tricky beast internally. Yay!

The next steps will be a function for consuming a verifiable stream and placing the received data in a byte storage. This one is more complex, because it needs to verify data as it arrives. Next, we will need a way of populating a byte storage by feeding in raw string data, which internally is then augmented with all required verification metadata. And finally, we need a nice way of exposing access to all of this: the storage trait only provides storage space, but does not track which parts of it have already been written to. Efficiently keeping track of that is the final unsolved design question in this stage of the work (I already know how to tackle this conceptually, but the details need more thought still).

~Aljoscha

This week I was attending Splintercon. This conference was focused on the notion of the ‘splinternet’, the fragmented result of the internet being regulated and censored by nations and corporations alike. Who benefits, and who suffers? Was this outcome inevitable?

There was always something maniacal about the late-90’s vision of the internet breaking down all distances, physical or cultural. I remember various graphics of world maps being collapsed into a single point, eradicating any notion of space and geography. This belied the reality that the internet is inescapably physical, a sprawl of cables and humming machinery forced to creep over the surface of the earth.

The idea that the very real differences between places would simply yield to the internet’s border-dissolving power was naïve at best. No technology survives contact with a culture. The car — another distance-collapsing technology — has found its way to every corner of the earth, where every place has developed their own regulations and culture around driving.

So perhaps the ‘splinternet’ was inevitable. Today that splintering comes from the top, from new legislation over the storage of personal data, to the outright censorship of information.

But why should governments and corporations be the only ones able to decide how networks are run? If the internet is splintering, we should all be able to get in on the fun. Maybe the problem is that the internet isn’t splintering enough.

~sammy

I want to share an application idea I have been dreaming up: the Cobweb. The Cobweb is a hypertext system with four distinguishing characteristics:

First, it has a built-in operator for transcluding multiple documents at once. Think like an iframe in HTML, but for a whole directory, or even multiple directories accross domains. With this functionality, you can replicate design patterns such as microblogging timelines or nested reply threads, while requiring neither ad-hoc server-side logic nor ad-hoc client-side logic. Everything simply arises from the document markup. I have written a blog post exploring this approach more thoroughly here.

Second, by virtue of being built on Willow — hyperlinks take the form of Willow URIs, of course — the Cobweb provides mutability without link rot. We need to normalise such systems.

Third, extreme simplicity. The complexity of the web necessarily concentrates power in the hands of browser vendors. I consider it imperative for alternative systems to pursue a contrary approach without compromise. The markup of the cobweb is essentially a subset of markdown, together with a bulk transclusion operator. The only inline media beyond text are images; only a single, simple image format is supported. Arbitrary files can be added as attachments at any position in the text, but without any special rendering logic beyond, effectively, a download button.

Fourth, efficiency. Markup is stored in a computer-friendly format (you might edit things as markdown, but that markdown would be saved in a binary format) — possibly even designed so you can mmap documents into memory and then work with them directly. The image format employs a form of interleaving that allows resource-constrained clients (or clients in low-bandwidth environments) to download only prefixes of images, which can then be rendered as low-quality versions of the final image. Bab ensures the integrity of the prefixes. Compression everywhere, though always preserving simplicity and thus ease of implementation.

The end result would be fairly constrained compared to the www, yet still intriguingly powerful — especially with bulk transclusion. And I envision extremely performant and low-memory-footprint Cobweb browsers, which would still manage to make everything really slick: great typography and customisable styling, inline-editing and authoring, real-time updates of transclusions, excellent UX around access control, storage size management, and so on.

Inspiration includes Gopher and Gemini, Man Pages and IETF RFCs (seriously, just imagine having something like the Cobweb instead of these — it keeps getting better the longer you think about it!), Mastodon, TiddlyWiki, and Joplin.

~Aljoscha

Last Sunday we received the news that our proposed talk for FOSDEM had been accepted. I’ll be presenting “Willow - A family of peer-to-peer storage protocols” in the Local-first, sync engines, and CRDTs track. I’ll do an overview of the Data Model, Meadowcap, etc, but I’d like to also get into some of our motivations, and maybe we’ll have some surprises too.

The last time I did a talk like this was in the Summer of 2023, at Offline in Berlin. We’d just started working on Willow, and we thought it wouldn’t be too long before we had a working implementation. Innocent days.

Looking back two and a half years later, without complete implementations of everything, is bittersweet. I recognise all the cul-de-sacs we wandered down, the missteps we made, and how I’d do things differently if I could go back in time. But in order to become the me who can recognise my own failures today, I had to go down all those dead ends first with my own two feet. There was no other way available.

So in the end I’m just grateful I get the opportunity to still do the things I wanted to do, even if it’s taken longer than I’d hoped. On many levels.

However, I do not want to get up on a stage and talk about Willow again, and then tell interested folks that they need to wait a bit longer: we’re aiming to have implementations of persistent Willow storage and the drop format completed by the time I head to FOSDEM.

ngl it’s going to be a little crunchy. To have persistent Willow data storage, we need persistent Bab, so this week I implemented a filesystem backend for Bab, implementing Bab’s ByteStorage trait over a thin wrapper over async-fs’s File. Aljoscha did a great job of designing this trait, I needed to know next to nothing about how Bab works to get this thing working. Ignorance truly is bliss.

Ping me If you’d like to meet up at FOSDEM!

~sammy

About two years ago, we published willowprotocol.org, and our server took a million requests on the first day. Soon we were getting emails from companies, universities, weird individuals, and it felt like we were really making something that made people sit up and take notice. We received our funding to build a rust implementation of Willow, and then… things stalled. It got quieter and quieter, until I imagine to the outside it seemed like nothing was happening at all. A lot did happen, of course, just not in a way we’d choose to have things happen again.

The end of 2025 brings the close to a weird era of willow, an era whose weirdness I’d like to reflect upon as I think I can recognise some of the ways it went differently than we’d hoped.

Our first mistake was maximalism. Willow Confidential Sync was the newest member of the Willow family, with the most novelty and flash. Range-based set reconciliation. Privately determining common interests between peers. Teeny tiny privacy-preserving message encodings.

As a poster child of privacy and efficiency, Confidential Sync radiated complexity onto everything around it. Willow storage needed to be able to efficiently fingerprint and partition ranges of local data for range-based set reconciliation. Storage needed special push-centric APIs so that new data could be broadcast to peers. Ufotofu needed a menagerie of encoding traits to be able to accommodate the private encodings which Confidential Sync uses to mitigate man-in-the-middle attacks.

We started to recognise the weight of this complexity early on, and specified the Willow Drop Format just a few months after the launch of willowprotocol.org. It’s a vastly simpler spec which asks far less of its dependencies. But at that point our funding goals and our destination were locked in: Confidential Sync was where we were going. And that path led us straight into the swamp.

At the same time, we were refactoring willowprotocol.org to use a new version of Macromania, the system we use to author the entire site (and this one!). Aljoscha made a big effort porting all the existing content to the new version, and it was so much nicer to author pages with that we couldn’t really go back to doing it the old way. And now we had a branch of the website with the new, not-quite-ready system and new content all mixed together. This interdependency congealed into a kind of seal which stopped us from making small updates to willowprotocol.orguntil everything was ready in one giant mega-update.

So willowprotocol.org was not getting updates. Which not only made it look like we were inactive to the outside world, but it also hurt our morale because we felt like we were digging into a bottomless pit with no recognition for our efforts. The longer the silence became, the worse our perfectionism got: we wanted to have something to show for ourselves after that long period of silence.

So we found ourselves working in frustrated silence amidst a myriad of interlocking dependencies. It was gruelling. In a struggle to keep the project on track, I ended up breaking personal connections which I treasured. There was also life outside of Willow which insisted on happening.

So what did we learn? Avoid complexity. Deliver frequently. Focus on doing as little possible to achieve your goal. Communicate often. How often have you heard these things? How many dozens of blog posts and presentations have you seen stressing these very points ad nauseam? I had certainly heard those hard-won lessons countless times — but found out I had to win them the hard way for myself, nevertheless.

I also have anti-takeaways: we were led into a swamp, but it was a very fruitful swamp. We used lots of time to make difficult, foundational design decisions, decisions which are already giving us increasingly stable foundations to work upon. It was long and tedious but it was also instructive, and we have the opportunity to actually use that experience.

worm-blossom.org is our antithesis to the era described in this editorial: frequent, ad lib, joyous. That’s what I want to bring more of to 2026!

~sammy

What a year. Sammy looked back, and I will lay out some future plans — specifically, plans for Willow. By the end of January, we want to have a simple persistent storage implementation and a Willow Drop Format implementation. Obviously, we have many more plans. Here they are, roughly in the order in which we want to make them real.

With Meadowcap, persistent storage, and the Willow Drop Format implemented, it will be possible to build actual sneakernet-based applications. While it is tempting to add more powerful data synchronisation as the next step, we should probably revisit end-to-end encryption of entries and payloads first. The earlier we implement this, the fewer moving pieces we will need to change based on our learnings. The concrete scope of this will consist of specifying specific techniques and primitives as part of Willow’25, and then to augment our storage implementations with the ability to encrypt and decrypt data on the fly. Ideally, this work will introduce only small changes to our public APIs — but ensure that most everything going forward is encrypted by default. Realistically speaking, a whole bunch of data is going to end up on the servers of large cloud providers, after all.

Before I can describe the next endeavours we have planned, I need some slightly abstract groundwork: the distinction between pull-based and push-based APIs. In a pull-based API, the entity that wants data needs to actively ask for it. In a push-based API, a data provider can proactively send data to its subscribers as new data arrives. Push-based APIs have several advantages, but they are also significantly more complex to get right. To a large degree, I blame the subpar progress we have made in the past on prematurely trying to nail the perfect push-based APIs: the Willow Confidential Sync protocol for replicating data, as well as ambitious subscription-based APIs for interacting with local data stores. This is why we have adopted a pull-based-only approach for now; this decision has played a significant part in our frequency (and hopefully quality) of recent releases.

Keeping this push-vs-pull distinction in mind, the next logical step in making Willow more “real” is a synchronisation protocol over the internet. The Willow Confidential Sync protocol is deeply push-based, so it is not an option. Instead, we have concrete plans for a more HTTP-inspired protocol of relatively simple requests and responses. This is the Willow Transfer Protocol (WTP), and you can read a draft version here. Completing the WTP specification and implementing it will be the next larger chunk of work, and one I am looking forward to!

Once we have a WTP implementation, we will have a complete (pull-based) Willow stack. It will then be time to slowly move to more efficient push-based components. The first step there will probably be local APIs. More specifically, a Bab implementation that allows subscribing to new incoming data, and then a Willow storage implementation that allows subscribing to change events. This will be crucial for more advanced applications that want to dynamically maintain custom indexes for more efficient queries than simply interacting with a Willow storage directly. In the long term, I actually hope that Willow stores will be used for replication and as a foundational source-of-truth exclusively, whereas any data access would go through more appropriate special-purpose indexes. Subscribable stores are the key stepping stone for moving toward that world.

Another step we will need to take eventually is to provide a more efficient storage backend. Right now we are building simplistic, linear-time storage that simply scans through all data and filters out unwanted results. But in the mid-to-long-term, we will need a proper three-dimensional data structure to provide access in logarithmic time. I have thoughts already around a three-dimensional generalisation of zip-trees (and, possibly, G-trees). But these details are a topic for a different time.

Speaking of improved local data access: we currently provide only embedded Willow storage, but long-term we would like to move to an actual database server with which multiple applications can interact concurrently (using capabilities to scope access rights, of course). The basics of this are fairly straightforward, but the challenging part is making push-based APIs work. Ideally, we'd like applications to be able to maintain persistent subscriptions even across several sessions. This would require the database server to maintain an event log, and applications would supply an offset at which to resume their subscription upon connection. When an application falls too far behind, it would catch up via set reconciliation. This sounds nice in theory, but the actual specification and implementation process will probably be rather thorny.

And after all of that, there is Willow Confidential Sync. I assume that we will learn enough from the WTP to warrant reworking the confidential sync spec. And then we will need to implement it, of course. Which feels a bit daunting right now, yet also exciting — confidential sync is were so many of our design decisions converge into a package that will likely be greater than the sum of its parts. I want to live in the world where you can integrate that package into your application designs with miniscule effort.

Other Willow-related plans include benchmarking suites, transactional APIs for our local data storage, a peer-sampling service (possibly in concert with confidential sync), and implementing the Willow URIs spec. Then there are application ideas — Rummager, the Cobweb, perhaps libraries of common UI components. And beyond Willow, we still want to release Macromania, and we have too many ideas for other side-projects to even start enumerating them here.

This is already far more than we can hope to accomplish in 2026. And we’ll doubtlessly accumulate even more ideas along the way. 2026 will be a busy year, but also a fun one, and one brimming with potential.

~Aljoscha

It was time to take a pause. In the last week I've spent time with family, ate, took my customary New Year's Eve bath, ate, and started a new journal for 2026. What little worm-blossom related stuff I did was thinking and sketching out my FOSDEM talk, specifically thinking about what you should say to an audience who have just seen twenty presentations on similar projects for the past several hours. Something high energy, controversial, more about vibes than details?

In honesty I am bracing myself for another intense year, perhaps harder than the last. So I am looking for ways to make it easier on myself. I found the practice of trying to make a funny comic each week a challenging one (with questionable results). I envy Aljoscha's free-form noodling in his little musical corner. So I'm going to have a little experiment with some more improvisational visual storytelling.

I hope you took some time to catch your breath over the last few weeks too, let's try our best!

~sammy

Friday arrived so quickly, and I had not written an editorial yet. I have barely thought about the website update at all this week, actually. Does this mean I finally found a healthier attitude toward my work and this website, will I now lead a life of increased inner harmony and peace? Will my output dwindle as a consequence?

Nah, I simply spent most of the time learning and thinking about audio file formats and video file formats, in anticipation of the Cobweb and Rummager.

The number of file formats you would need to support to provide somewhat decent coverage for the media content on the web is intimidating, as is the staggering complexity of the individual file formats. So I have been pondering what a significantly more simple stack could look like. Only one file format per media type (images, sound, video), and simple formats at that.

The file formats would be pure delivery formats, no niceties for editing included. Further, the formats should support both lossy and lossless compression (which, sadly, rules out the Quite Okay Image format and the Quite Okay Audio format), down to extreme measures such as one-bit images or audio. And, ideally, the three formats would have identical or at least similar structure, and reuse the same underlying techniques. A nice challenge would be to fit all three formats (including their shared components) into 400 * 3 = 1200 lines of C code — the reference implementations of QOI and QOA take 400 lines of code each, after all.

A rough starting point for a unified approach to the three different file formats:

• Regard each media type as a collection of channels, where each channel is a list of integer samples. For images, the four channels are three colour channels (YCbCr, specifically) and the alpha channel, and the one dimension is the row-wise content. For audio, each channel (one for mono, two for stereo, possibly more for more fancy stuff) is a stream of PCM data. For video, one channel encodes time-warying images, and the other channels are audio channels.
• Channels are interleaved. Each channel has an independent frequency. For example, a video could encode images at 30 Hz but audio at 44.1 kHz. And an image could use lower frequencies for the colour channels (aka chroma subsampling).
• Before encoding, quantise all signals to a channel-specific bitwidth. This is the (only) part where lossy compression enters the picture. The bitwidth of a channel can even be zero — say, for an alpha channel that has the same value everywhere. The simplemost quantisation scheme is simply to right-shift each sample by some amount, though for audio it is tempting to go with a non-linear quantisation scheme (because using fewer bits for high frequencies compared to low frequencies neatly exploits properties of human hearing).
• On the quantised signals, use some form of Lempel-Ziv compression. When a value cannot be reconstructed from the dictionary, it is not encoded verbatim, however, but as a residual from a prediction (see next bullet point).
• Each file format has a (different kind of) predictor, which tries to predict the next sample from the previous samples. We never encode samples verbatim, instead we encode the difference to the predicted value (the residual). With a good predictor, this means we will usually encode only small numbers, a property we can then exploit for more compact encodings. For images, the prediction is probably simply the previous sample. For sound, I’d tend toward reusing the LMS of the QOA format. And for video image frames, the pixel of each frame would be predicted as being the pixel at the same position of the previous frame, offset by a frame-specific x and y value (i.e., simple motion compensation, I'd also consider simple forms of Dolly and camera rotations if they admit simple integer-based implementations).
• Finally, encode the residuals or the LZ compression stuff nicely and compactly. The easiest approach here would be to define a bit-based encoding and simply encode every sample separately, but it would also be nice to have a byte-based encoding — and grouping multiple samples together can be more efficient than encoding them individually. So there is a lot of design space here. Further, there are concerns around seeking: how can you efficiently jump to an arbitrary point of time in a time-varying signal? First, there probably needs to be a balance between absolutely encoded samples and relatively encoded samples. You'd use the absolute samples for seeking, and then do a linear scan across the remaining relative samples. The original anti-monotone binary linking scheme would be a really neat way for providing (only a bounded number of) forward-pointers between absolute samples. But still, with interleaved channels of different frequencies and bit-depths, and LZ compression, finding a good mechanism here is tricky.

There are further concerns such as container formats, and accessibility features like subtitles. But this would be the general idea for building an image format, an audio format, and a video format that all work in essentially the same way. It might not be unrealistic to use identical subsampling, quantisation, LZ compression and final encoding steps for all three formats. Aside from working out the specifics, the big remaining question is whether the lossless compression yields acceptable file size reductions compared to raw data, and whether the quantisation step (which is the only form of lossy compression in the whole approach) yields sufficient quality for the selected degree of compression. The two Quite Okay File formats make me fairly optimistic in that regard, however.

~Aljoscha

My week was high-intensity but low-worm-blossom as I was single parenting. With the time I had between biking my kids to school through the snow, preparing meals, pleading with them to just open their mouth so I could brush their teeth... I mulled over my FOSDEM presentation.

I got stuck in a 'concept' phase. For me it would be a bit disappointing to just say what Willow is and explain it spec by spec. After all, we have a website for that. This presentation should act as a complement to it. But how? I sat around pondering. Thinking really hard. And guess what: it didn't help.

As soon as I switched to making — even with only a very vague idea — things began to progress. Being able to sketch, respond, tear down, iterate. Share an idea I'm excited about with Aljoscha, and he gets excited too. And now I can't wait to keep working on this thing.

And so the first thing we did was confirm with the devroom organisers that we would be able to play audio over the speakers.

~sammy

Rambly, reflective editorial incoming — creativity is fascinating (as is reading McGilchrist).

I was almost completely unable to make any progress coding this week, despite desperately wanting to. But whenever I opened the text editor, I would just be completely disinterested. I’d like to tell myself that I have the cognitive ability and trained skills to be able to work productively even when I’m not into it. But truthfully, the code I did force myself to write is so bad I will probably have to rework it at some point.

I usually struggle with creative output because I judge my own intuition too harshly and want to get things right. This website has been a good exercise in loosening up in that respect. But this week, the problem was of a different nature — I did not care enough to judge at all, leading to pretty bad code I normally would not let myself write. So where ideally there should be a healthy balance of generative creativity and quality control, this week had neither. And forcing things regardless just doesn’t work.

And then, out of nowhere, my brain delivered the solution for how to elegantly implement verification of incoming data streams in Bab. And suddenly, I felt like coding again, and have been happily coding since. This did not happen in time for finishing things by today, so I will detail this implementation approach next week.

Interestingly enough, while I was more or less unable to code, I did do some beepbox pieces I am not unhappy with after the fact — both of which I imagine working nicely in an RPG of some sorts, inspired by watching some Expedition 33 and Final Fantasy VII over the week:

And I did not dread the task of writing an editorial for this week either, despite having little idea what to write about before actually sitting down and starting to type. Not sure whether this says something about myself, or more about differences between writing, music, and coding.

~Aljoscha

This week my attention was once again fully occupied by the upcoming FOSDEM talk. After a run-through of a draft slide deck this morning, I felt dissatisfied, although everything was arguably there. Empty calories.

I am a person driven by dissatisfaction, and I work on things until I am happy with them. Aljoscha has the same malady. There is seemingly no cure. This makes the things we work on take a very long time. This website is a kind of compromise: we blog about our ongoing dissatisfaction on a weekly schedule.

So now I am restructuring the presentation for what is hopefully the final time. I mentioned last week that if you want to learn about what Willow is and how it works, we have a website. But why is Willow like it is? What kind of world made us feel such a thing was necessary? That is the kind of talk I think people will enjoy most.

It will be fun, maybe even insightful. If you have been reading worm-blossom.org for any amount of time, how this presentation turns out will not surprise you in the least.

~sammy

(also, Bab implementation notes)

~Aljoscha

I’m writing this on the second leg of my seven hour train journey from the Hague to Basel. Like Aljoscha, I’m going to p2p-basel to listen rather than to speak. Tags: receptive, curious, critical. And frequently: confused. Plenty of leaning over to Aljoscha and whispering “what does that mean?”. I still remember him explaining monotonicity to me a few years ago with the patience of a saint (St. Numbers).

The FOSDEM talk’s shaping up rapidly. I’ve now updated its title and description to better reflect its contents: “Willow - protocols for an uncertain future”. This is going to be a very different take on Willow than we’ve presented before, so I hope it will be interesting even if you’ve been hanging around the project for a while.

This week, I want to start adding more random little one-off drawings to our weekly updates.

~sammy

I have been programming too much over the past weeks; it has been impoverishing my life. Case in point: tomorrow (I am writing this on Wednesday), I am leaving for p2p-basel, and I... have barely thought about that at all, being too busy doing computer things. I don't even have a presentation to give, whereas in the previous years my problem usually was having too many presentation ideas to choose from. Still, there is much to look forward to: I will be seeing old friends again, will meet new interesting folks (there'll be someone from the Pijul VCS, I might finally get closure on some question regarding this paper), there will be interesting discussions and fun evenings. And I'll be meeting Sammy in person for the first time in over a year!

Despite me programming too much, we will not get everything ready by FOSDEM that we had hoped to. Which makes me feel disappointed, despite the fact that I have actually been pretty productive: Bab slice verification is implemented, and I am close to having a simple storage backend that can store a single, contiguous slice. In terms of unlocking funding milestones, I am more productive than needed. Yet the self-imposed FOSDEM-deadline makes me feel bad about the current state of things.

Hence, I label the self-imposed deadline silly, and I will do my best to focus on everything I did implement instead of what is still missing. As usual, there is also the voice that wants Willow to finally become a real thing that people use. But if listening to that voice means wasting my time staring at a small rectangle all day, then I label that voice silly for now as well.

In less silly news, academia and macadamia have a Levenshtein distance of two. Hmmmm, macadamia... now that is something I can get behind!

~Aljoscha

So, P2P Basel happened. I had a nice and calm train ride there together with Andreas and Glyph from p2panda, where we avoided talking about p2p stuff for the most part, knowing the weekend would hold more than enough of that. I did get a bunch of music recommendations from Andreas.

The evening had a couple of workshop participants meet for lunch. I had a nice chat with Pierre from Pijul, who gave me some much needed reassurance that most everyone has moved on from their phd thesis topic by the time it comes down to writing, and also that most everyone hates publishing by that time. The key is to do it anyway, I guess. So for the first time in well over a year, I've actually been motivated to keep chipping away at this phd thingy.

Then Friday was the first actual day of the workshop. Except I didn't feel terribly well. I made it through the day, but ended up shambling back to the apartment with a fever. And then I spent the remaining weekend lying in bed, with Sammy bringing me the occasional meal from the venue.

I'm still recovering, so no implementation progress report this week. Though I am a bit concerned how long it has been since the last actual code release. I hope I'll get over the hump of releasing the first storage-backed-Bab version soon, so that things can get back to more regular, incremental feature releases.

~Aljoscha

Back at last. And I owe the faithful readers of worm-blossom.org not one… but two editorials, as these last two weeks I was busy attending two very different events: p2p-basel and FOSDEM. The former is intimate and deep, the latter huge and sprawling. And I had a great time at both. Let’s start with Basel.

p2p-basel invites researchers and p2p practicioners to a weekend of lectures and talks at the University of Basel. This year there were only approximately 16 participants, and that creates a very intimate atmosphere for discussion. For example, the catering of the event is also peer-to-peer, with attendees participating in cooking and cleaning. Those moments where you stand shoulder to shoulder with someone while peeling carrots present wonderful opportunities for deep discussion.

There are prepared presentations and a time dedicated to spontaneous unconference discussions, of which I moderated two this year.

The first was on ‘delay tolerant UIs', anticipating user confusion stemming from systems where authored data may only reach other users a week later. This might mean the arrival of a message divorced from its original context, or worse, placed into a new context which changes its meaning (e.g. a thumbs up emoji arriving during the discussion of something sad). Clearly we cannot simply lift the UIs of applications which presume near-instant data delivery and expect them to work.

The second was on ‘zero / low identity systems’, and I was really happy that so many others were also interested in this topic. Digital identity is often used to link different pieces of data together through a ‘digital double’ which can often be linked to a real person. The risk here speaks for itself, and I had a half-baked idea on how to reduce or eliminate identity in networks: random IDs for all data, with the option for the ‘owner’ of a random ID to derive a new one which everyone else can then verify is provably linked to the original ID. This forms a kind of ‘identity on demand’, or as someone else put it much more eloquently, ‘lazy identity’. It was pointed out that this is not conceptually unlike image board tripcodes (though I’ve since learnt these are apparently quite crackable). Maybe there’s something here, though? It would be interesting to plug such a system into Willow.

Huge thanks to p2p-basel’s organiser Erick, who has relentlessly improved the event year after year.

~sammy

Weekly updates and depressive episodes are not a good match.

I'm trying to take good care of myself, as much as I am able to. Even if that means taking a break in the exact week where FOSDEM has hopefully pulled a lot of fresh eyes toward us.

I often feel like Willow is coming into being too slowly. But even then, we are still doing meaningful and creative work, and putting it out for everyone to interact with. And that is something I can draw strength from, even when I am too off-balance to meaningfully act on that strength.

~Aljoscha

A handful of days after returning from Basel, I departed again for FOSDEM in Brussels. This was my first time attending, and I was a little overwhelmed by the size of the event. So many buildings, people, and cat ears. And for some reason I chose to wear heels the whole weekend.

Luckily I was met by friends of worm-blossom (FoWB) Miaourt and danmw, and we spent Saturday discussing blobjects, fictional Willow companion hardware / tamagotchis, radio-friendly Willow, and Miaourt bearing a seemingly inexhaustible bag of waffles. We had a lovely worm-blossom.org dinner that evening, and although I did not say it at the time because I wanted to appear cool and not overly sappy, I thought to myself that if these were the kinds of people worm-blossom is interesting to, then surely we are doing something right.

I spent Sunday in the Local-first, CRDTs and Sync Engines devroom. My favourite talk was from Modal Collective, in which they detailed their motivation for building a new secure, iPhone-quality OS with p2p at its heart. It shared a lot of themes with my own presentation and I was excited that this line of thinking is turning into action. It’s always lovely to see adz of p2panda, and lovely to meet Tobias Bernard of GNOME too.

Then it was my turn to present. I’ve been working on my talk since the start of the year, and it had been through several incarnations before reaching its current one, leaving me a week and a half to produce approximately fifty drawings. Fortunately I embraced the scrappy spirit of worm-blossom.org, and put together possibly the only talk of FOSDEM with an opening cutscene (music provided by Aljoscha, naturally):

(There is also a live recording if you’d like to experience the live vibes and Q&A portion).

Readers, four days later I am still glowing from the reception to my talk. I’m so, so grateful for all the kind things people said to me afterwards, and the lovely conversations it afforded later that evening.

As I took the train home, I thought about how the p2p community has grown over the years, and how the connections between us have matured alongside the theory and implementations. Perhaps we can do something really wonderful together?

~sammy

The music for the intro of the FOSDEM presentation is a beepbox piece, of course:

~Aljoscha

Shortly after last week’s update my body — sensing uncharacterstic relief — instantly succumbed to a flu that I’m still not completely rid of. So I spent a lot of time in bed reading The Rose Field and quietly feeling pleased with myself about my FOSDEM talk doing the rounds. Again, thank you for watching (and maybe even sharing) it, the response has meant a huge amount to me.

With no more slides to prepare, I returned to programming (shudder). It has been a while, and I feel clumsy and slow, especially in Rust. But I’ve started implementing the prerequisite encodings for Willow’s drop format. These use Willow’s private encodings, where items can only be successfully decoded with a bit of common knowledge. This might sound like overkill for a non-interactive method of exchange like the Drop format, but as Aljoscha reminded me: “turns out trying to omit as much data as possible results in short codes”. And what if that makes all the difference for someone to squeeze as much Willow data possible on their low-capacity USB key? That is what I tell myself as I port existing implementations to another, hopefully final incarnation of an Ufotofu-based encoding abstraction.

I’ve also been thinking about Discord, which will reportedly soon start requiring age verification to access all features. I belong to a few communities based on Discord which will sadly disperse if this ends up happening. And it doesn’t seem like there’s any persuasive answer for where to disperse to. Why is that? Is it because the pieces (protocols, people) aren’t there yet? Or is there a deeper problem in that even if we had all those pieces, fusing them into a monolithic whole requires an amount of coordination (and funding) we collectively can’t muster? I have included some links in this week's update to ponder.

~sammy

This week saw a fruitful discussion about Ufotofu, triggered by Dan’s implementation of chaining combinators. In a nutshell, the problem is that some methods irretrievably swallow up their arguments in case of an error.

There is an obvious fix, which is to have the error case return not only the actual error but also the original arguments. We should be able to apply this for bulk processors, but things are less clear for regular consumers: if consume returned the item it was supposed to consume in case of an error, then that would break the symmetry with the producer design.

For a terrible day-and-a-half or so, I was actually ready to break the symmetry here, in order to make the chaining combinator work without cloning (with a destructive consumer API, the only sensible workaround is to clone items before feeding them into the consumer, so that you still have the clone around after an error). But then I thankfully realised that everything works nicely for types that are Copy. So we can have “normal” APIs that require their items to implement Copy, and then do more expensive and clunky variants that require only Clone.

In any case, the discussion with Dan helped clarify a couple of muddy aspects around ufotofu, and has given me a new perspective on what makes Copy so special in the context of ufotofu items — Copy exempts types from linear typing, thus making them match my purely mathematical mental model of the design space.

~Aljoscha

Recently I’ve felt like a weird little frog who is finally finding their weird little microhabitat. The perfect temperature! Exactly the kind of slime I like! Other weird frogs like me! January put me in the path of lots of people who do and make things I’m excited about, and who seem excited about the work I’m doing too.

The only problem being that I actually have to do that work.

Our immediate goal with Willow is to implement the means to save and retrieve Willow data, and to have at least one way to exchange that data. For the latter, we’re going with the Willow Drop Format, which collects lots of Willow data into a single file that you can deliver with whichever way you like to send files.

Since I started working on private encodings last week, we’ve merged implementations of private path encoding, private area encoding, and private authorisation token encoding to willow_rs’s main branch. I had to get all these past the relentless scrutiny of our fuzz testing apparatus (and Aljoscha), so I’m fairly confident these new implementations are robust and well documented. Which feels great! There is definitely some kind of maladjusted satisfaction I get from merging code, which is something I should probably watch out for.

With that out the way I can get back to the important work of being a weird little frog. This weekend I’ll be heading to the Internet Archive’s new European Headquarters in Amsterdam for a little borrel, where I guess I’ll do borrel-y things like eat little cubes of cheese... and with any luck, meet some more frogs.

~sammy

I finally made good progress on the persistent backend of the Bab implementation. All the logic is in place; the remaining steps are to rearrange the public API, document it, and then to add more thorough tests.

And instead of writing a long editorial, I’ll continue programming now.

~Aljoscha

When last week's update went up, I was at the Internet Archive Europe HQ in Amsterdam ('HQ' makes it sound like a big shiny building. It's really a cute little house by a canal). Remember last week when I said I felt like a frog finding their special frog habitat? Well this was more frogs. Many people who I'd been looking forward to meeting, and (somewhat amazingly to me) a fair number who'd heard of Willow before. After a lot of conversation, a sensible amount of cheese, and maybe a little bit of schmoozing, I left feeling energised. For years I've been trying to find a local network of people interested in this niche field, and either I was too much of a shut-in (likely), or this community in the Netherlands has finally reached a kind of critical mass and level of interconnectedness with the broader network.

Meanwhile, I worked on willow_rs. Unfortunately (and kind of intriguingly) my work finally presented a dead end in how well Rust is able to accommodate Willow's generic and parametrised ambitions. Aljoscha has (hopefully) done a write up on this below (don't leave me hanging aljoscha). I've started working on an implementation of the Willow Drop Format (WIP pull request here), where I am practicing liberal use of the todo! macro to implement as much as possible without letting little things like not having that dependency yet weigh me down. This is like taking a breadth-first approach where instead I've always taken a depth-first approach to missing bits and pieces.

~sammy

What is going on in Blossoquest? Since when did it even have that name? Why are we suddenly in a Western? Does the crack in the wall have any meaning? How can Sammy claim dislike for Homestuck and unfamiliarity with Hussy’s work, and yet the third panel looks like Problem Sleuth and the name like a play on Bard Quest? What is going on?!

Anyway, apparently I am supposed to write about the limitations of the Rust type system instead of investigating these mysteries. So here we go!

The Willow specs are highly generic. So naturally, our Rust implementation is full of generics as well. Because this is rather cumbersome, we are also providing a non-generic implementation of Willow’25 for application developers who shouldn’t need to care about the flexibility offered by the parameterisability of Willow.

In our previous codebase, the willow25 implementation merely provided type aliases, with the consequence that compiler messages bombard developers with all those hidden generics. So in the reboot of the codebase, we went for proper wrapper types. We even have a wrapper! macro that automatically generates the structs that wrap the underlying generic types. So far, so good.

The first problem that stems from this arrangement is that of converting between references. Suppose our generic implementation has a function that requires a &Entry<Lots, Of, Generics>. To implement the willow25 version of that function by calling the original function, we need to convert &Willow25Entry into a &Entry<Lots, Of, Generics>. Reference-to-reference conversions are not exactly the most idiomatic Rust. But this was doable, with the `wrapper!` macro even completely hiding the required unsafe and #[repr(transparent)] shenanigans that make it work. And up until Sammy’s recent implementation work, this was good enough.

The real trouble started however once we needed willow25 wrappers for generic types made up from other generic types. For example, a generic AuthorisedEntry struct consists of a generic Entry struct and a generic AuthorisationToken struct. A Willow25 wrapper around AuthorisedEntry would internally still use generic components. But for some code, we would need it to consist of (or at least to be interpreted as consisting of) a non-generic Willow25Entry and a non-generic Willow25AuthorisationToken. But for this interpret-the-generic-components-as-being-non-generic-wrapper-types business, the generics must be instantiated with types that satisfy certain constraints regarding their memory layouts. The exact details are gnarly and do not really matter, the important part is that we ended up in a situation where there would be unsafe code galore. This in itself is bad but not a showstopper, but the real problem was that the safety of this unsafe code would depend on whether you instantiated the generics in a certain way.

The compiler has no way of enforcing any invariants here. And I have never seen any code whose safety depended on properly instantiating type parameters. Calling this situation unidiomatic would not do it justice, I’d rather reach for an unholy abomination whose creators should be barred from writing Rust code ever again.

So what now? We will keep the generic code base for now, but new work will only provide willow25 APIs. Where possible, those will still make use of the underlying generic implementation, but where that does not work we will simply duplicate the generic code and then rip out the type parameters. Long-term, we will probably phase out the generic code base completely, doing concrete willow25-based implementations directly instead.

This is a bit of a shame. The genericity of the specs is a decision I still stand by, and generic implementations do have a certain elegance. But it turns out that (idiomatic) Rust is just not particularly well-suited to it. And we want to keep the codebase accessible and high-quality. The decision to move to non-generic implementations for now stings, but everything else would lead to an idiosyncratic code base that would have made Dominic Tarr proud, and scared off everyone else.

(Also, writing docs for two almost identical versions of our APIs was a pain from the very start, so I cannot truthfully report that I am entirely unhappy.)

~Aljoscha

Let’s start with nuts and bolts dev diary stuff: I started working on a new persistent store for Willow. It’s a straightforward port of the previous persistent store, which used sled under the hood. Except things were just different enough that I could justify switching to Fjall instead. Translating from sled’s APIs to fjall’s was pretty straightforward! This store is missing a few pieces still, as the Store trait needs to be updated with a few methods for storing and retrieving payloads, which depends on the Bab work Aljoscha has been feverishly boshing away at. So: we’re getting there!

While that bubbles away, what else to do but learn about Tauri, and seeing what I can glue together with our Willow Rust APIs? I got about as far as creating one button before my head filled with question about how we’re going to present all of this to human beings. Like the keypairs which are used to sign entries, for instance. they’re just not anything like user accounts on something like Discord, or even Mastodon.

Since the earthstar days, we’ve referred to the set of features keypairs confer as an ‘identity’. But to me, identity smacks of some kind of vague proving humanity rubbish. But that’s kind of exactly the opposite of what we want Willow’s keypairs to do:

• they can be used to prove authorship
• but you can have a lot of them
• and they have as much or as little identifying information attached to them as you want.

So how about ‘persona’? Data can be verified to be written by a persona, you can have as few or as many as you like, and you can attach as much ‘identity’ information (e.g. display name, avatar) you want… or not. Miaourt suggested using little masks as the iconography for this, which I really love.

I'm not the only working on Willow GUIs right now, so expect more of this!

~sammy

So much has happened this week! This calls for bullet points!

• Good progress on the Bab implementation: we have well-documented public API for (persistent) single-slice-per-string storage with support for partial verification. Still requires better tests, but then it can be released. Before that, I’m creating a Bab-aware trait for Willow stores in the willow25 crate though, to unblock Sammy’s implementations of a persistent store and the Willow Drop Format.
• I attended a DWeb preparation event on Saturday. Was nice to run into many familiar faces, and to get to know some new ones. Went home with renewed motivation to get Willow out into the world.
• I met with Zelf to record an episode of her podcast. It will probably be a while until editing is finished and this episode is released, though.
• On Tuesday, a bunch of us p2p nerds ended up in a bar with Olivia Jack of Hydra fame. Hydra is a super cool environment for creating 2d graphics — basically a synthesizer for graphics. I may have spent too much time playing around with it afterwards. It is so easy to get something fun going. I also may have started reading the source code and begun sketching a WebGPU-based implementation. The chances for that ever getting anywhere near useable are close to zero, but at least I had a very fun day of rest.
• I have strong opinions on variable-length integer encodings. So I was happy when I read Brooklyn Zelenka’s new bijou64 spec, which is an interesting variation on my older VARU64 specification. Slightly more complicated, but also slightly more efficient, and arguably easier to detect non-nanonic encodings. An interesting read for the other three people who care about varints.
• This week I finished a super secret side-project that I will reveal... later :>

~Aljoscha

I’ve had repetition on my mind this week, most explicitly triggered by Lu Wilson’s writing on the subject, but also earlier through some reflection on the music I’ve been making. My commentary for last week’s background music track was simply I feel like I'm repeating myself too much with these =/. By now, I’m much more willing to embrace the repetition.

Later, I went to a free improvisation concert played by Fred Frith and Liz Albee, and the weakest passages by far were those where they did not feel comfortable to simply repeat themselves.

And later still, a book I’m reading had the following A. N. Whitehead quote:

[..] no rhythm can be a mere pattern; for the rhythmic quality depends equally upon the differences involved in each exhibition of the pattern. [..] A mere recurrence kills rhythm as surely as does a mere confusion of differences.

I don’t think this makes a lot of sense, at least not when applied to music. No sound happens in isolation, everything is shaped by the sounds that came before. Even when repeating a pattern verbatim, the effect on the listener does not repeat: the first time you hear it, it is new. The second time you hear it, the impression drastically changes, because it is not new. The third time, it is a repetition of something that was already familiar. And so on.

Each repetition has a different quality, because it was preceded by a different context. The quality of change arguably decreases (the difference between the first and the second repetition is more pronounced than that between the 50st and the 51st repetition), but it keeps changing. After 400 repetitions you get a completely different effect than after 32 repetitions. Just like you cannot step into the same river twice, you cannot truly repeat the same pattern.

Oh, wait, this is supposed to be a dev diary, right?

Our Bab implementation now has a well-documented public API, and Sammy has started coding against it! I will spend the upcoming week writing more thorough tests, and next Friday I think we will release it properly. Writing tests is well and good, but I need to accept that I’ll release buggy libraries anyway. So might as well put a disclaimer on the release and ask for bug reports.

~Aljoscha

Early this week I made a few wireframes for a kind of generic Willow management UI (with Penpot, which is nice).

Discord’s ‘server’ UIs have always mapped very cleanly to Willow’s namespaces, and it seems negligent not to appropriate their work. But as soon as you start wading into the more unfamiliar concepts like keypair and capabilities the difficulty begins to reveal itself.

One of the most beautiful ideas about systems like Willow is using a single namespace as a bucket for all kinds of data. What if your Discord server also stored a wiki, a webpage, or anything else its users wanted to put into it? What if you could easily inline your wiki pages into your chats, and weave your data together? And had decentralised permissions which could stay on top of this menagerie of data?

Do you interface with that data in a single mega interface, or many seperate ones? If the latter, does each interface manage its own personas and passes (keypairs and capabilities)? In that case, do you reveal those nitty gritty details like being granted access to /chat/general or something aware of its context, like “General discussion”?

We've bonked our heads against this many times over the years. We tried a reusable React component that you could drop into your (presumably React-powered) app. We tried a kind of web-based mini-OS with different interfaces running inside a managed UI frame. And now I am just starting to think that the solution is a set of agreed conventions, spanning from representation to UI. I uttered the cursed phrase “cultural API” earlier, but I suspect this might just be a rubbish way to say pattern language.

Next week I’ll make an appearance at XPUB (a two-year master in experimental publishing in Rotterdam) to talk about p2p, Willow, and how we communicate our work. I’m really looking forward to this, and I’m challenging myself to find different ways to tell a story about peer-to-peer and Willow. For this one, it’s going to be a timeline, thirty years of lessons learnt the hard way. I feel like it would be irresponsible to only conclude we need to march on with grim hypervigilance. Surely we’re guarding something precious with that vigilance? It’s not just our safety, but something precious, joyful, and unrealised?

~sammy

This Monday I took the train to Rotterdam to go speak at XPUB.

Before we started, Doriane (who had invited me after we’d met at FOSDEM) gave me a quick tour of the XPUB facilities, way up on the top floor of a building overlooking the Nieuwe Maas. I was really enamoured with the atmosphere of their space, which had zine-filled shelves, boards crowded with posters, and students working away in cosy, personal spaces. It didn’t at all have the feeling of a curated display case, but rather a living, breathing space with work flowing through it. Nice.

And as for my part: we neatly avoided a sammy slideshow situation (thank you doriane) and instead had a kind of guided discussion for two hours, in which we hardly talked about Willow. It was great. I had a big chalkboard behind me on which we assembled a timeline of nearly three decades of peer-to-peer trials and tribulations, from Napster’s legal dismantling to the state-sanctioned deactivation of Apple’s Airdrop. Interjections from the rest of the room were frequent and always thoughtful, and when we finished at 19:30 it felt like we could have kept on going.

Like I said last week, I couldn’t leave it on a downcast note. After all, what are we going to all this trouble for? I argued that we had an opportunity to remember that data is just a byproduct, and that the point is for it to change us somehow. Peer-to-peer protocols have some unique features which encourage us to learn to become conscious stewards — rather than hoarders — of data.

Anyway. Great time, great people, hope to be invited back again some day.

But that’s not all I got up to this week. I also implemented payload storage in the new Willow persistent store, using the Bab implementation Aljoscha has been working on. Streaming, verified storage! It’s a beautiful thing. Or it will be a beautiful thing once we’ve got all the bugs ironed out.

~sammy

Do I write a thoughtful editorial, or do I simply finalise this rather late update? While I have no idea about the identity of the BOTTLENECK in Blossoquest, I sure know that for this week’s update I was the bottleneck. But maybe there is time for sharing a few fun ideas still.

1. A live-coding environment for shaping the sound of digital instruments. Basically, you edit a function that takes as inputs a frequency, a volume level, the time since the note was triggered, and the time since it was released, and you return a value between -1 and 1. This function is called 44000 times per second to produce sound. Oh, and the function has access to the samples it produced for the past x seconds, so that you can employ feedback and self-reference in shaping the sound. (shoutout to the best repo on github, but this would have a live-refreshable js interface instead of compiling rust programs into command line utilities)
2. A theremin-like, mouse-based (or touch-pen!) UI for playing the live-coded instruments. Should have a toggle for switching between continuous frequencies and a mode that discretises into the 12 notes of well-tempered tuning. And while you move the right hand on the frequency-volume domain, the left hand can do key-presses: one key functions as a sustain pedal, another represents whether to produce sounds at all (pressing and holding that key is like pressing and holding a note on a piano).
3. A language for writing down instructions for triggering the digital instuments. Basically, what happens if you take the good parts of Lilypond notation and free it from the complexities of score engraving. You end up with a language for structuring events in a flow of time. The language would “compile” into a stream of MIDI or OSC signals. And, the theremin-like UI has a recording function that saves live input in the form of that written language.
4. Also, for something completely different: cellular automata (think game of life), but with a notion of continuous force fields. Cells can emit a force field, and the values of the field instantaneously propagates between cell updates, with a configurable decay based on distance to the emitter(s). Imagine a tree-drawing automaton where each cell of tree trunk acts as an emitter on the force field of treeness, and when picking a direction for growth, cells of low treeness are more likely to be picked.
5. And how do I turn the oscillations of a cellular automaton into music?

ALSO: I have taken some work-in-progress drafts of other ideas of mine and moved them from obscure branches into a work-in-progress directory here at the deployment of worm-blossom.org. There is an incomplete but already informative spec draft of the media file formats I wrote about some weeks ago. There is a sketch of a bit-level variable length integer encoding for usage in the media formats. And there’s a draft of the website for Macromania. Without styling, and without most of the content, but the getting-started guide and the introduction-to-macros writeup are functional. You could start using Macromania today!

Oops, I suppose I did end up writing a full editorial.

~Aljoscha