“Do geese see god?”

2025Week 4510 Nov

ufotofu

We have released the ufotofu crate, a Rust library for working with streams and sinks, which will power our Rust implementations of Willow. Because ufotofu will be a quite foundational building block, we put a lot of effort into documentation, including a somewhat over-the-top website. Please check it out!

Rust already has a de-facto standard library for async sequence processing, so we do not necessarily expect wide adoption for ufotofu. But we believe that some of its designs are genuine improvements over many libraries out there across all programming languages. So we hope that awareness of our little crate will spread, and that it will influence the design of other libraries in this domain. The ufotofu website goes into all design choices in great detail, and is interesting not only to Rust programmers but to API designers in any programming language.

Special thanks go to Glyph, for suggesting to use a palindrome as the crate name, and for early implementation work.

Have you opened the ufotofu website yet?

A drawing of Sammy, arms crossed with a pencil tucked behind her ear.

Last week I was chasing funds for the next generation of worm-blossom projects. One of these proposals is a general-purpose GUI application called Rummager. In our original application I breathlessly wrote about "organically expressing your own applications through the act of nesting free-form data". But what does that actually mean, you (and potential funders) ask? A picture's worth a thousand words, and I've combined several into a single image below which is a lot of economy.

We also put together the new ufotofu site. Ufotofu provides a fairly low-level abstraction, and I think we'd be forgiven for providing a link to a README and calling it a day. But to be earnest for a minute: right now we have funding which gives us a lot of freedom. We probably won't have that funding forever. When it's over, I want to look back at what we did with this opportunity and be satisfied we did our best.

That means we want to describe ufotofu really well. We want to make that description pleasant to read. And we want to add touches that make a reader want to stick around. Those final touches may seem frivolous (we made it so that when you scroll it looks like a UFO is beaming up stuff), but I think (hope) readers see frivolity and recognise it as care.

~sammy

A friendly computer waving his mouse, beckoning you to come contribute!!!

Contributions Welcome: `ufotofu` crate (Rust)

https://codeberg.org/worm-blossom/ufotofu/issues

A new crate, plenty of new opportunities to contribute. We have curated a selection of issues, many of which are good opportunities to get to know ufotofu better without being too demanding in terms of Rust knowledge or time investment.

And over in the willow_rs repository we have a fresh issue which is relatively simply to implement and touches on willow_rs, bab_rs, and ufotofu. Fun!

A drawing of Aljoscha, fingers interlaced, eager to deliver some interesting news.

I got into distributed systems through Secure Scuttlebutt (SSB). A lot of what happened during my first years in that community has stayed with me until today. In many cases, that is a good thing. In other cases, things are more debatable. My stance on libraries for streams and sinks might be one of those latter cases.

SSB was created by Dominic Tarr. He was well-known in the early node-js community, amongst other things for his pull-streams library. Node-js had its own set of stream abstractions, but Dominic judged he would be more productive if he built SSB on a custom library. And he was probably right. In the early years, people where pretty happy to code against a sane API, it was only years later that the non-standard streaming API became an obstacle for new contributors.

Then, there was André Staltz, who had created cycle.js and later also xstream and callbag. And when Cryptix and Keks implemented SSB in Go, the first thing they did was to build a custom streams library. Patchwork was built on a custom UI framework, built on custom observables.

So for some reason, I ended up absorbing knowledge from a bunch of people who were not afraid of rolling and re-rolling their own stream libraries. And over the years (starting, probably, around 2017), I began forming my own opinions on this matter. At the last few iterations of p2p-basel, I always ended up frantically scribbling on whiteboards to convey the isomorphism between a good stream abstraction and regular grammars. When we started Rust work for Willow, I somehow managed to convince Sam to use part of that funding for writing a new library of stream abstractions in Rust. We also managed to convince NLNet to fund this work. I submitted an essay about stream APIs to Onward! 2024 (which got rejected, but with some very helpful feedback), which has formed the basis of the ufotofu webpage.

I’m rambling. Ufotofu has been simmering for almost ten years now. Over that time I have changed as a person, as a computer scientist, as an engineer. The designs of ufotofu have also changed, but the core principles have always remained the same. I think what I am trying to say is that I am proud of this thing.

I’ve seen many people criticise Dominic for implementing SSB on top of a custom streaming abstraction. Calling it inaccessible, accusing him of not-invented-here syndrome. But I think Dominic made a pragmatic choice: he built his own tools, because he could make them better than the tools that were already around, and the task ahead of him was sufficiently large to justify the initial overhead. We are limited by the quality of our tools, after all. And I feel similarly about ufotofu. Which is why I am ignoring the lessons I perhaps should have learned from SSB. Willow in Rust is implemented on ufotofu, and as of now, I fully believe that to be the right choice.

~Aljoscha

2025Week 4403 Nov

“existence is inefficient”

what's with the name?!

why is this website called ‘worm-blossom’? In this series we will explain why in a concise manner.

Part 1: The French Revolution

France, 1792. The monarchy has been abolished and the French Republic established. Anything seems possible, and everything is up for reconsideration. After all, if you can overthrow a centuries-spanning monarchy, what can’t you do?

Revolutionary ideals, scientific vigour, and a bureaucratic need to govern a fledgling republic have brought about a metric revolution. Variable, localised, and unpredictable systems of weight and measure are being replaced by the gram and metre. These easily divisible units lubricate France’s scientific, military, political and economic progress.

Could the same by done for time? Jean-Charles de Borda, a military engineer and scientist thought so. By dividing the day into ten hours, the hour into 100 minutes, and each minute into 100 seconds, he argued that timekeeping itself could be revolutionised.

By 1793, the National Convention had formally adopted this system. Decimal time was used in all public records, and a number of decimal clocks were prominently displayed in a few French cities.

But mandatory use of decimal time was suspended just two years later. It didn’t seem to offer much of a benefit, nobody was going to enforce its use, and a lot of people felt that a decimal hour (equivalent to 2.4 regular hours) was just too long for an hour. Not to mention how annoying it was going to be to replace every watch and clock in France.

Decimal time would remain largely forgotten for another 203 years, which is where this series will be continued. See you next time!

New `willow25` Crate Version

We have released version 0.2.0 of the willow25 crate. This release adds the groupings module, which implements our groupings specification (who would have thought!)

There is a type for each kind of grouping: WillowRange for ranges, Range3d for 3d ranges, Area for areas, and AreaOfInterest for areas of interest.

As with the previous release, we tried to make working with these groupings as convenient as possible. For example, you can use the same method on entries to check for membership in any kind of grouping (a marvel powered by the Grouping trait):

use willow25::prelude::*;

let namespace_id = NamespaceId::from([0; 32]);
let subspace_id = SubspaceId::from([1; 32]);

let entry = Entry::builder()
    .namespace_id(namespace_id.clone())
    .subspace_id(subspace_id)
    .path(path!("/blog/ideas"))
    .timestamp(1793)
    .payload(b"Temps décimal! Très bien!")
    .build().unwrap();

// Range creation from arbitrary Rust ranges!
let time_range = TimeRange::from(..=2025.into());
assert!(entry.is_in(&time_range));

let area = Area::new(
    None, path!("/blog"), time_range,
);
assert!(entry.is_in(&area));

See the new groupings tutorial for more examples of what these APIs feel like, or read the API docs.

I accidentally published an incredibly nerdy but surprisingly useful Rust crate this week: order_theory.

The Willow specs often refer to minimal and maximal values (such as SubspaceIds or Paths). Sometimes we also need a notion of successors on these types. So I defined some traits capturing these concepts. And then I moved them into their own crate, because I needed them in several places. And since you cannot very well publish a trait claiming to provide order-theoretic concepts without traits for lattices, I added a corresponding hierarchy of traits as well.

Then I needed our NamespaceId and SubspaceId types to implement SuccessorExceptForGreatest. So I added a generic implementation of all order-related concepts of our new crate for arbitrary arrays, and simply delegated to those from willow25.

Next, I wanted to convert arbitrary Rust ranges (which feature both inclusive and exclusive bounds) to Willow ranges (which are less flexible). Turns out that you can do those conversions exactly if the type in question is totally ordered and has a notion of predecessors and successors. Good thing I had recently defined traits for exactly that!

All Willow groupings support a notion of intersections — the intersection of two groupings is the grouping which contains exactly the entries contained in both original groupings. Why, that sure sounds like a LowerSemilattice. How convenient I had a trait for that concept.

Finally, implementing intersections (and other order-related methods) is not always easy. So I added utilities for property testing to the order_theory crate. And these then found several errors in our previous Willow groupings implementations.

Math! Yay!

~Aljoscha

Not much to report this week as I was travelling. The coming week is going to be busy as November is here, and 2026 is starting to rear its fuzzy little head. I think it's going to be a good year, but only if I write all the proposals and applications like I'm supposed to.

~sammy

2025Week 4327 Oct

“ideally, do the max”

A comic.

In the first panel, Anubis, Egyptian God of the Afterlife, addresses a man:

The Big Giant Willow Website Update

willowprotocol.org, the online home of our peer-to-peer storage protocols, has been updated to reflect all of the work that we’ve done on Willow from April 2024 until now (work funded by our friends at NLnet! Hartelijk dank!). That’s eighteen months of work in a single serving! Argh!!! So what are the highlights?

Firstly, we’ve reworked Confidential Sync (neé W.G.P.S) to be man-in-the-middle attack resistant. In the old protocol, if two peers who did not know each other before were to sync over a compromised network (e.g. a network with a router modified to intercept traffic), an attacker would have been able to access all sorts of Willow-stored data! As Confidential Sync is expressly designed for secure syncing between untrusted peers, we've redesigned Private Interest Overlap to mitigate this kind of threat. You can learn more about the security properties and threat model of this design here.

The Drop Format (neé Sideloading protocol), which encodes a user-selected subset of Willow data into a single binary (ready to put on a USB key or send via Signal), now is able to add entries without their payload. We’ve also made the encoding more efficient so you can squeeze even more Willow data into a smaller number of bytes.

We’ve added a new page for Willow‘25, a recommended set of efficient, secure parameters for all the Willow specs. This way, nobody needs to witness the nightmare of generic types we’ve created for ourselves unless that’s what they really want.

There’s a new Rust section which has tutorials for the crates we’ve released so far. Not only does this section have a completely gratuitous illustration of Willowverse mascots Alfie and Betty riding a giant crab, but we’ve also integrated rustdoc support into willowprotocol.org’s referencing system. This means that our tutorials can reference things like structs and their methods, and link readers directly to up-to-date Rust API documentation. We will be regularly updating this section over the coming weeks and months!

There are new illustrations, diagrams, and comics pretty much everywhere. The Drop Format spec. The Private Interest Overlap Page.Encrypting Entries. The ‘More’ section. The About us page.

Another page which has lots of new illustrations is the front page, which we completely redesigned to make answering the question of “what is Willow?” a bit easier. It also has a big fun illustration for desktop users.

All of this work got stuck in a kind of mega branch comprised of 67,020 additions and 14,590 deletions over 330 commits with the sole description of ‘do not merge yet, wip’. No comment.

~sammy

Shout-out!

Many thanks to Miaourt for their contributions to Bab — which in turn made it into today’s willow25 crate release! Miaourt and I (Aljoscha speaking) did a pair coding session on Friday, which was quite fun. And a good reminder that I am merely an academic dabbling in coding, whereas there are people who have a lot more practice and are actually good at it. I was really impressed with how quickly they grasped everything, and they made several choices that resulted in much cleaner code than what I would have written.

`willow25` Crate Release

We have released the first version of our Rust implementation of Willow’25. This initial release implements the full Willow Data Model, with the parameter choices recommended in the Willow’25 spec.

Highlights include

an optimised immutable Path struct with zero-copy prefix creation,
a Timestamp struct allowing for convenient math such as some_timestamp - 3.seconds(), and
the Entry struct which will sit at the core of all Rust Willow applications.

We put a lot of care into making everything as ergonomic as possible. Look at the following snippet — isn’t it neat? We think it is neat!

use willow25::prelude::*;

// Create an entry for a specific payload,
// timestamped with the current time.
let entry = Entry::builder()
    .namespace_id([0; 32].into())
    .subspace_id([1; 32].into())
    .path(path!("/vacation/plan"))
    .now().unwrap()
    .payload(b"See many sights!")
    .build().unwrap();

// Create a new entry based on the previous
// one, with greater timestamp and a new payload.
let updated = Entry::prefilled_builder(&entry)
    .timestamp(entry.timestamp() + 5.minutes())
    .payload(b"See *all* the sights!")
    .build().unwrap();

// Assert that the new entry would overwrite
// the old entry if both were inserted in a store.
assert!(updated.prunes(&entry));

Accompanying this release is a brand-new Rust section on the Willow website with tutorials for our Willow crates. Check it out, even if only for the fun illustration!

There might exist gifted programmers who can write non-trivial code and get things right the first time. I am not among them. But still, there is no way around implementing everything for a first time.

Over the past year, we have implemented most everything we specify on the Willow website in Rust. For the first time. Technically, that code does what it is supposed to do. But a lot of it is cumbersome to use, and there are several structural problems I really don’t want to get into (here). Long story short: we are reworking that code, and re-releasing it over the coming weeks. And from what we can tell so far, the new code is pretty enjoyable to work with!

We started this week by releasing an implementation of the Willow data model. Here is the plan for what will follow:

Groupings (a requirement for any non-trivial Willow usage),
Meadowcap (at that point, you can start sketching out access-control for Willow-based applications),
traits for (persistent) storage for entries and payloads (at that point, you can write full application logic against the traits),
simple in-memory and persistent implementations of the store traits (at that point, you can write and run full offline-only applications), and
the Drop Format (at that point, you can write and run full sneakernet applications).

Once these are done, we will probably slow down a bit, and play around with our own application ideas, end-to-end encryption, network-based synchronisation, and anything else that might come up. In addition to working on Bab, Ufotofu, and Macromania, that is.

~Aljoscha

Shout-out!

Some weeks ago, Joakim, emailed us to point out that UNIX timestamps have less than great behaviour when it comes to leap seconds, and recommended International Atomic Time (TAI) as a better alternative. We have indeed updated the recommended interpretation of the timestamp in Willow to be a TAI timestamp.

And then over the past days, Joakim supplied us with many valuable details around the specifics of choosing a reference epoch (i.e., “which instant does the timestamp zero denote”), saving us from making a non-standard choice due to some confusing (and possibly buggy) documentation and code in the Rust hifitime crate. This might even help those crate developers to fix some independend bugs of their own. Quite a journey — thank you for your help!

Contributions Welcome: `Willow25` crate (Rust)

https://codeberg.org/worm-blossom/willow_rs/issues

Want to help us improve the willow25 crate? We have collected some opportunities, from simple standard-library trait implementations over instructive Path mangling to rather advanced but kinda fun nice-to-haves.

2025Week 4220 Oct

“visceral ambivalence”

A comic with four panels.

The first panel's text reads:

The Surfers’ Review

“fantastic site wow!!”

~ Dante Scanline

“Always excited to see what yourself and Aljoscha are up to, often outside my ability to understand but always filled with such an incredible feeling of hope”

~ Dook

“The humor and art work are also very pleasant.”

~ moid

We thank you for your warm reception to the first instalment of worm-blossom.org.

This week I put a lot of time in non-willow projects. The biggest timesink was ufotofu, which is a library for working with lazy sequences (“sinks” and “streams”) in Rust. Specifically, I wrote tons of documentation, tests, and documentation tests in preparation for a proper release. This crate has served us well for our async needs while implementing Willow in Rust, and I am giddy about getting to share it more widely soon!

I also put some time into Macromania, our typescript-embedded templating language we use to generate willowprotocol.org in all its hyperlinked and tooltippy glory. We have accumulated a couple of breaking changes we wanted to do to the core of Macromania, and those are now all implemented. Sadly, this means that all the fancy macros powering tooltips etc do not work with that new version of Macromania.

As for the current state of Macromania: the core library is now stable and has pretty solid doc comments (the curious can look at the jsr release), but we still lack high-level documentation for onboarding folks into the madness. We will start working on a proper website featuring this high-level documentation soon-ish (for me, ufotofu has higher priority at the moment). That website might turn out fairly unimpressive initially, because we still lack all the really powerful macros — not only did we break their foundation, but we have also accumulated a bunch of breaking improvements we want to roll out for most of our core macros. Nevertheless, the new version of Macromania already builds this website, and does some fun stuff such as generating an RSS feed and highlighting code snippets. See the source and tinker around if you are feeling adventurous!

This was a highly productive week from my side for ufotofu and Macromania, yet I feel twinges of guilt. Both of these projects are fairly computer-centric, not human-centric. Well, ufotofu in particular; it tries to apply some mathematical principles to streamline a programming experience. Programming for programming’s sake, as if the world could seriously be improved by making computers and programming more efficient and thus even more entrenched... But still, I cannot deny that this kind of work is extremely fun and satisfying to me. In weeks like this, I carry with me some cognitive dissonance centered around escapism and indulgence, counteracted by telling myself that all of this keeps me motivated and will ultimately further Willow (not only by keeping me engaged, but also because our Willow implementations rely on ufotofu and our web presence relies on Macromania).

I want to end on a more light-hearted note, so, erm, have some borderline hyperactive sketches for two pianos (loosely inspired by listening to some Meredith Monk pieces):

Also, *how* is “instalment” correct spelling? ~_~

~Aljoscha

Frugal Async Rust

A whole lot of the Rust code we write for Willow is async, and async Rust comes with some pitfalls. We have decided to make some simplifying assumptions to avoid the worst stumbling blocks. And those assumptions, we have now put on a webpage for others to reference. Frugal Async Rust describes how we forego certain features but in exchange avoid whole worlds of pain.

And we have released a frugal_async crate with some basic utilities adhering to the frugal async principles, for example, an awaitable Mutex:

use frugal_async::mutex::*;
use core::ops::{Deref, DerefMut};
use core::time::Duration;
use smol::{block_on, Timer};

let m = Mutex::new(0);
block_on(futures::future::join(async {
    let mut handle = m.write().await;
    // Simulate doing some work.
    Timer::after(Duration::from_millis(50)).await;
    assert_eq!(0, *handle.deref());
    *handle.deref_mut() = 1;
}, async {
    // This future is "faster", but has to wait
    // for the "work" performed by the first one.
    let mut handle = m.write().await;
    assert_eq!(1, *handle.deref());
}));

We encourage everyone who enjoys the details of async Rust to look at this issue around creating a macro for repeatedly calling several async methods in a loop and acting on the results as they become available.

This week I have been mostly single parenting, so I did not get much programming done. Most of my creative energies went towards cooking meals.

For breakfast I often turn to the humble porridge. I like to put big chunks of apple in my porridge which cook just long enough to get soft but still keep their structure, kind of like in an apple pie. I like to cook my porridge low and slow, which really annoys everyone else.

You can't go far wrong with a curry. I break up a cauliflower into little florets and roast them in the oven until they start to caramelise at the edges. I could eat them just like that, but in a curry sauce they're even better. I make a curry sauce from scratch and it is completely inauthentic.

Salmon is great when you're low on time because you just smack it in the oven for a bit. In the colder months I like to do a roast vegetable mix alongside it: potato, beetroot, sweet potato, red onion, topped with some capers. The purple of the beetroot and onions look so good next to everything else.

And when it all became too much for me I just ordered pizza. hashtag winning

~sammy

And then, inspiration struck...

2025Week 4113 Oct

“we've done the first 99%, now we only need to do the other 99%”

Spindly tunnels wind down deep underground, branching off to aborted ends, and at the very bottom, a cavern with two twits inside. One asks the other: Why are we down here, anyway?

A New Era Dawns

For the past year, we have been pretty busy here at worm-blossom. We have also been pretty quiet. But that is about to change. From now on, you can expect weekly updates about our progress on this very website. As you can probably tell already, we will restrict our reports to the minimal technical details, the bare necessities, the austere metrics of productivity. Nothing else.

Throughout our work on Willow, we increasingly had to take on technical debt in order to meet deadlines. Combined with our inclination of keeping work to ourselves until it is just perfect, we ended up accumulating a vast body of work we stayed fairly quiet about. Not only did this make us look stagnant, it also made us unhappy. But now, with our previous grant obligations fulfilled, and the next grant hopefully leaving us more breathing room, we decided to take action. Going forward, we will

identify minimally useful subsets of everything we do and publish these as early as possible,
apply this policy not only to future work, but also to our backlog of pretty cool accumulated things,
emphasise the exuberance and creative spark that drives us to do our work,
invite and mentor contributions to our projects, and
share our progress in the most joyful way we can on this website.

And what better way of holding us accountable to these lofty resolutions than to boldly and pinkly announce them on the world wide web?

~Aljoscha

`bab_rs` Crate Release

We have released the first version of our Rust implementation of Bab. Bab is a family of hash functions inspired by BLAKE3, specifically designed for content-addressed storage systems. Read more about Bab here.

The bab_rs crate provides both a fully generic implementation (the Bab specification is parameterised over choices such as digest sizes) and a ready-to-use implementation of WILLIAM3 (our recommended default parameterisation).

While we have plans — and funding — for some sophisticated features around verified streaming, for now the crate provides the basic API you would expect for incremental hashing: you create a Hasher, repeatedly feed it with bytestrings, and eventually call finish to obtain a digest.

use bab::Hasher;

// Create a hasher.
let mut hasher = Hasher::new();

// Feed data into the hasher.
hasher.write(&[0, 1, 2]);
hasher.write(&[3, 4]);

// Obtain the WILLIAM3 digest.
let digest1 = hasher.finish();

assert_eq(digest1, [
    57, 194, 179, 8, 66, 14, 197, 192,
    5, 56, 128, 49, 249, 78, 60, 119,
    231, 114, 96, 94, 138, 132, 41, 172,
    54, 42, 99, 229, 59, 43, 122, 225,
]);
// Yay =)

This week we renamed two Willow specs: Willow General Purpose Sync and the Sideloading Protocol. I think it's important to sit up and notice when every time someone hears the name of your project, they say "like GPS??? Huh???". And at this point the term 'sideloading' has such a toxic association with locked down mobile platforms that it felt like a completely unforced error.

We renamed them to Confidential Sync and Drop Format, respectively. Not only are these nicer names, but it also meant I could redo the emblem for the WGPS.

A photograph of some sketches of the new Confidential Sync emblem

The WGPS was the only spec which didn't have a nature-inspired emblem, and this has now been remedied. I also remedied the total absence of pink from the lineup. It's a grapevine, because with confidential sync you "heard it through the grapevine". Isn't that precious?

~sammy

Contributions Welcome: Add a `Digest` type (Rust)

https://codeberg.org/worm-blossom/bab_rs/issues/1

Our Bab implementation currently uses simple arrays for hashes. It would be nice to have a dedicated Digest type with features such as constant-time equality comparisons, zeroing on drop, hexadecimal printing, and integration with the digest crate.

“Do geese see god?”

ufotofu

Contributions Welcome: ufotofu crate (Rust)

“existence is inefficient”

New willow25 Crate Version

“ideally, do the max”

The Big Giant Willow Website Update

Shout-out!

willow25 Crate Release

Shout-out!

Contributions Welcome: Willow25 crate (Rust)

“visceral ambivalence”

The Surfers’ Review

Frugal Async Rust

“we've done the first 99%, now we only need to do the other 99%”

A New Era Dawns

bab_rs Crate Release

Contributions Welcome: Add a Digest type (Rust)

Contributions Welcome: `ufotofu` crate (Rust)

New `willow25` Crate Version

`willow25` Crate Release

Contributions Welcome: `Willow25` crate (Rust)

`bab_rs` Crate Release

Contributions Welcome: Add a `Digest` type (Rust)