worm-blossom

Let’s start with nuts and bolts dev diary stuff: I started working on a new persistent store for Willow. It’s a straightforward port of the previous persistent store, which used sled under the hood. Except things were just different enough that I could justify switching to Fjallinstead. Translating from sled’s APIs to fjall’s was pretty straightforward! This store is missing a few pieces still, as the Store trait needs to be updated with a few methods for storing and retrieving payloads, which depends on the Bab work Aljoscha has been feverishly boshing away at. So: we’re getting there!

While that bubbles away, what else to do but learn about Tauri, and seeing what I can glue together with our Willow Rust APIs? I got about as far as creating one button before my head filled with question about how we’re going to present all of this to human beings. Like the keypairs which are used to sign entries, for instance. they’re just not anything like user accounts on something like Discord, or even Mastodon.

Since the earthstar days, we’ve referred to the set of features keypairs confer as an ‘identity’. But to me, identity smacks of some kind of vague proving humanity rubbish. But that’s kind of exactly the opposite of what we want Willow’s keypairs to do:

• they can be used to prove authorship
• but you can have a lot of them
• and they have as much or as little identifying information attached to them as you want.

So how about ‘persona’? Data can be verified to be written by a persona, you can have as few or as many as you like, and you can attach as much ‘identity’ information (e.g. display name, avatar) you want… or not. Miaourt suggested using little masks as the iconography for this, which I really love.

I'm not the only working on Willow GUIs right now, so expect more of this!

~sammy

So much has happened this week! This calls for bullet points!

• Good progress on the Bab implementation: we have well-documented public API for (persistent) single-slice-per-string storage with support for partial verification. Still requires better tests, but then it can be released. Before that, I’m creating a Bab-aware trait for Willow stores in the willow25 crate though, to unblock Sammy’s implementations of a persistent store and the Willow Drop Format.
• I attended a DWeb preparation event on Saturday. Was nice to run into many familiar faces, and to get to know some new ones. Went home with renewed motivation to get Willow out into the world.
• I met with Zelf to record an episode of her podcast. It will probably be a while until editing is finished and this episode is released, though.
• On Tuesday, a bunch of us p2p nerds ended up in a bar with Olivia Jack of Hydra fame. Hydra is a super cool environment for creating 2d graphics — basically a synthesizer for graphics. I may have spent too much time playing around with it afterwards. It is so easy to get something fun going. I also may have started reading the source code and begun sketching a WebGPU-based implementation. The chances for that ever getting anywhere near useable are close to zero, but at least I had a very fun day of rest.
• I have strong opinions on variable-length integer encodings. So I was happy when I read Brooklyn Zelenka’s new bijou64 spec, which is an interesting variation on my older VARU64 specification. Slightly more complicated, but also slightly more efficient, and arguably easier to detect non-nanonic encodings. An interesting read for the other three people who care about varints.
• This week I finished a super secret side-project that I will reveal... later :>

~Aljoscha

When last week's update went up, I was at the Internet Archive Europe HQ in Amsterdam ('HQ' makes it sound like a big shiny building. It's really a cute little house by a canal). Remember last week when I said I felt like a frog finding their special frog habitat? Well this was more frogs. Many people who I'd been looking forward to meeting, and (somewhat amazingly to me) a fair number who'd heard of Willow before. After a lot of conversation, a sensible amount of cheese, and maybe a little bit of schmoozing, I left feeling energised. For years I've been trying to find a local network of people interested in this niche field, and either I was too much of a shut-in (likely), or this community in the Netherlands has finally reached a kind of critical mass and level of interconnectedness with the broader network.

Meanwhile, I worked on willow_rs. Unfortunately (and kind of intriguingly) my work finally presented a dead end in how well Rust is able to accommodate Willow's generic and parametrised ambitions. Aljoscha has (hopefully) done a write up on this below (don't leave me hanging aljoscha). I've started working on an implementation of the Willow Drop Format (WIP pull request here), where I am practicing liberal use of the todo! macro to implement as much as possible without letting little things like not having that dependency yet weigh me down. This is like taking a breadth-first approach where instead I've always taken a depth-first approach to missing bits and pieces.

~sammy

What is going on in Blossoquest? Since when did it even have that name? Why are we suddenly in a Western? Does the crack in the wall have any meaning? How can Sammy claim dislike for Homestuck and unfamiliarity with Hussy’s work, and yet the third panel looks like Problem Sleuth and the name like a play on Bard Quest? What is going on?!

Anyway, apparently I am supposed to write about the limitations of the Rust type system instead of investigating these mysteries. So here we go!

The Willow specs are highly generic. So naturally, our Rust implementation is full of generics as well. Because this is rather cumbersome, we are also providing a non-generic implementation of Willow’25 for application developers who shouldn’t need to care about the flexibility offered by the parameterisability of Willow.

In our previous codebase, the willow25 implementation merely provided type aliases, with the consequence that compiler messages bombard developers with all those hidden generics. So in the reboot of the codebase, we went for proper wrapper types. We even have a wrapper! macro that automatically generates the structs that wrap the underlying generic types. So far, so good.

The first problem that stems from this arrangement is that of converting between references. Suppose our generic implementation has a function that requires a &Entry<Lots, Of, Generics>. To implement the willow25 version of that function by calling the original function, we need to convert &Willow25Entry into a &Entry<Lots, Of, Generics>. Reference-to-reference conversions are not exactly the most idiomatic Rust. But this was doable, with the `wrapper!` macro even completely hiding the required unsafe and #[repr(transparent)] shenanigans that make it work. And up until Sammy’s recent implementation work, this was good enough.

The real trouble started however once we needed willow25 wrappers for generic types made up from other generic types. For example, a generic AuthorisedEntry struct consists of a generic Entry struct and a generic AuthorisationToken struct. A Willow25 wrapper around AuthorisedEntry would internally still use generic components. But for some code, we would need it to consist of (or at least to be interpreted as consisting of) a non-generic Willow25Entry and a non-generic Willow25AuthorisationToken. But for this interpret-the-generic-components-as-being-non-generic-wrapper-types business, the generics must be instantiated with types that satisfy certain constraints regarding their memory layouts. The exact details are gnarly and do not really matter, the important part is that we ended up in a situation where there would be unsafe code galore. This in itself is bad but not a showstopper, but the real problem was that the safety of this unsafe code would depend on whether you instantiated the generics in a certain way.

The compiler has no way of enforcing any invariants here. And I have never seen any code whose safety depended on properly instantiating type parameters. Calling this situation unidiomatic would not do it justice, I’d rather reach for an unholy abomination whose creators should be barred from writing Rust code ever again.

So what now? We will keep the generic code base for now, but new work will only provide willow25 APIs. Where possible, those will still make use of the underlying generic implementation, but where that does not work we will simply duplicate the generic code and then rip out the type parameters. Long-term, we will probably phase out the generic code base completely, doing concrete willow25-based implementations directly instead.

This is a bit of a shame. The genericity of the specs is a decision I still stand by, and generic implementations do have a certain elegance. But it turns out that (idiomatic) Rust is just not particularly well-suited to it. And we want to keep the codebase accessible and high-quality. The decision to move to non-generic implementations for now stings, but everything else would lead to an idiosyncratic code base that would have made Dominic Tarr proud, and scared off everyone else.

(Also, writing docs for two almost identical versions of our APIs was a pain from the very start, so I cannot truthfully report that I am entirely unhappy.)

~Aljoscha

Recently I’ve felt like a weird little frog who is finally finding their weird little microhabitat. The perfect temperature! Exactly the kind of slime I like! Other weird frogs like me! January put me in the path of lots of people who do and make things I’m excited about, and who seem excited about the work I’m doing too.

The only problem being that I actually have to do that work.

Our immediate goal with Willow is to implement the means to save and retrieve Willow data, and to have at least one way to exchange that data. For the latter, we’re going with the Willow Drop Format, which collects lots of Willow data into a single file that you can deliver with whichever way you like to send files.

Since I started working on private encodings last week, we’ve merged implementations of private path encoding, private area encoding, and private authorisation token encoding to willow_rs’s main branch. I had to get all these past the relentless scrutiny of our fuzz testing apparatus (and Aljoscha), so I’m fairly confident these new implementations are robust and well documented. Which feels great! There is definitely some kind of maladjusted satisfaction I get from merging code, which is something I should probably watch out for.

With that out the way I can get back to the important work of being a weird little frog. This weekend I’ll be heading to the Internet Archive’s new European Headquarters in Amsterdam for a little borrel, where I guess I’ll do borrel-y things like eat little cubes of cheese... and with any luck, meet some more frogs.

~sammy

I finally made good progress on the persistent backend of the Bab implementation. All the logic is in place; the remaining steps are to rearrange the public API, document it, and then to add more thorough tests.

And instead of writing a long editorial, I’ll continue programming now.

~Aljoscha

Shortly after last week’s update my body — sensing uncharacterstic relief — instantly succumbed to a flu that I’m still not completely rid of. So I spent a lot of time in bed reading The Rose Field and quietly feeling pleased with myself about my FOSDEM talk doing the rounds. Again, thank you for watching (and maybe even sharing) it, the response has meant a huge amount to me.

With no more slides to prepare, I returned to programming (shudder). It has been a while, and I feel clumsy and slow, especially in Rust. But I’ve started implementing the prerequisite encodings for Willow’s drop format. These use Willow’s private encodings, where items can only be successfully decoded with a bit of common knowledge. This might sound like overkill for a non-interactive method of exchange like the Drop format, but as Aljoscha reminded me: “turns out trying to omit as much data as possible results in short codes”. And what if that makes all the difference for someone to squeeze as much Willow data possible on their low-capacity USB key? That is what I tell myself as I port existing implementations to another, hopefully final incarnation of an Ufotofu-based encoding abstraction.

I’ve also been thinking about Discord, which will reportedly soon start requiring age verification to access all features. I belong to a few communities based on Discord which will sadly disperse if this ends up happening. And it doesn’t seem like there’s any persuasive answer for where to disperse to. Why is that? Is it because the pieces (protocols, people) aren’t there yet? Or is there a deeper problem in that even if we had all those pieces, fusing them into a monolithic whole requires an amount of coordination (and funding) we collectively can’t muster? I have included some links in this week's update to ponder.

~sammy

This week saw a fruitful discussion about Ufotofu, triggered by Dan’s implementation of chaining combinators. In a nutshell, the problem is that some methods irretrievably swallow up their arguments in case of an error.

There is an obvious fix, which is to have the error case return not only the actual error but also the original arguments. We should be able to apply this for bulk processors, but things are less clear for regular consumers: if consume returned the item it was supposed to consume in case of an error, then that would break the symmetry with the producer design.

For a terrible day-and-a-half or so, I was actually ready to break the symmetry here, in order to make the chaining combinator work without cloning (with a destructive consumer API, the only sensible workaround is to clone items before feeding them into the consumer, so that you still have the clone around after an error). But then I thankfully realised that everything works nicely for types that are Copy. So we can have “normal” APIs that require their items to implement Copy, and then do more expensive and clunky variants that require only Clone.

In any case, the discussion with Dan helped clarify a couple of muddy aspects around ufotofu, and has given me a new perspective on what makes Copy so special in the context of ufotofu items — Copy exempts types from linear typing, thus making them match my purely mathematical mental model of the design space.

~Aljoscha

Back at last. And I owe the faithful readers of worm-blossom.org not one… but two editorials, as these last two weeks I was busy attending two very different events: p2p-basel and FOSDEM. The former is intimate and deep, the latter huge and sprawling. And I had a great time at both. Let’s start with Basel.

p2p-basel invites researchers and p2p practicioners to a weekend of lectures and talks at the University of Basel. This year there were only approximately 16 participants, and that creates a very intimate atmosphere for discussion. For example, the catering of the event is also peer-to-peer, with attendees participating in cooking and cleaning. Those moments where you stand shoulder to shoulder with someone while peeling carrots present wonderful opportunities for deep discussion.

There are prepared presentations and a time dedicated to spontaneous unconference discussions, of which I moderated two this year.

The first was on ‘delay tolerant UIs', anticipating user confusion stemming from systems where authored data may only reach other users a week later. This might mean the arrival of a message divorced from its original context, or worse, placed into a new context which changes its meaning (e.g. a thumbs up emoji arriving during the discussion of something sad). Clearly we cannot simply lift the UIs of applications which presume near-instant data delivery and expect them to work.

The second was on ‘zero / low identity systems’, and I was really happy that so many others were also interested in this topic. Digital identity is often used to link different pieces of data together through a ‘digital double’ which can often be linked to a real person. The risk here speaks for itself, and I had a half-baked idea on how to reduce or eliminate identity in networks: random IDs for all data, with the option for the ‘owner’ of a random ID to derive a new one which everyone else can then verify is provably linked to the original ID. This forms a kind of ‘identity on demand’, or as someone else put it much more eloquently, ‘lazy identity’. It was pointed out that this is not conceptually unlike image board tripcodes (though I’ve since learnt these are apparently quite crackable). Maybe there’s something here, though? It would be interesting to plug such a system into Willow.

Huge thanks to p2p-basel’s organiser Erick, who has relentlessly improved the event year after year.

~sammy

Weekly updates and depressive episodes are not a good match.

I'm trying to take good care of myself, as much as I am able to. Even if that means taking a break in the exact week where FOSDEM has hopefully pulled a lot of fresh eyes toward us.

I often feel like Willow is coming into being too slowly. But even then, we are still doing meaningful and creative work, and putting it out for everyone to interact with. And that is something I can draw strength from, even when I am too off-balance to meaningfully act on that strength.

~Aljoscha

A handful of days after returning from Basel, I departed again for FOSDEM in Brussels. This was my first time attending, and I was a little overwhelmed by the size of the event. So many buildings, people, and cat ears. And for some reason I chose to wear heels the whole weekend.

Luckily I was met by friends of worm-blossom (FoWB) Miaourt and danmw, and we spent Saturday discussing blobjects, fictional Willow companion hardware / tamagotchis, radio-friendly Willow, and Miaourt bearing a seemingly inexhaustible bag of waffles. We had a lovely worm-blossom.org dinner that evening, and although I did not say it at the time because I wanted to appear cool and not overly sappy, I thought to myself that if these were the kinds of people worm-blossom is interesting to, then surely we are doing something right.

I spent Sunday in the Local-first, CRDTs and Sync Engines devroom. My favourite talk was from Modal Collective, in which they detailed their motivation for building a new secure, iPhone-quality OS with p2p at its heart. It shared a lot of themes with my own presentation and I was excited that this line of thinking is turning into action. It’s always lovely to see adz of p2panda, and lovely to meet Tobias Bernard of GNOME too.

Then it was my turn to present. I’ve been working on my talk since the start of the year, and it had been through several incarnations before reaching its current one, leaving me a week and a half to produce approximately fifty drawings. Fortunately I embraced the scrappy spirit of worm-blossom.org, and put together possibly the only talk of FOSDEM with an opening cutscene (music provided by Aljoscha, naturally):

(There is also a live recording if you’d like to experience the live vibes and Q&A portion).

Readers, four days later I am still glowing from the reception to my talk. I’m so, so grateful for all the kind things people said to me afterwards, and the lovely conversations it afforded later that evening.

As I took the train home, I thought about how the p2p community has grown over the years, and how the connections between us have matured alongside the theory and implementations. Perhaps we can do something really wonderful together?

~sammy

The music for the intro of the FOSDEM presentation is a beepbox piece, of course:

~Aljoscha

So, P2P Basel happened. I had a nice and calm train ride there together with Andreas and Glyph from p2panda, where we avoided talking about p2p stuff for the most part, knowing the weekend would hold more than enough of that. I did get a bunch of music recommendations from Andreas.

The evening had a couple of workshop participants meet for lunch. I had a nice chat with Pierre from Pijul, who gave me some much needed reassurance that most everyone has moved on from their phd thesis topic by the time it comes down to writing, and also that most everyone hates publishing by that time. The key is to do it anyway, I guess. So for the first time in well over a year, I've actually been motivated to keep chipping away at this phd thingy.

Then Friday was the first actual day of the workshop. Except I didn't feel terribly well. I made it through the day, but ended up shambling back to the apartment with a fever. And then I spent the remaining weekend lying in bed, with Sammy bringing me the occasional meal from the venue.

I'm still recovering, so no implementation progress report this week. Though I am a bit concerned how long it has been since the last actual code release. I hope I'll get over the hump of releasing the first storage-backed-Bab version soon, so that things can get back to more regular, incremental feature releases.

~Aljoscha

I’m writing this on the second leg of my seven hour train journey from the Hague to Basel. Like Aljoscha, I’m going to p2p-basel to listen rather than to speak. Tags: receptive, curious, critical. And frequently: confused. Plenty of leaning over to Aljoscha and whispering “what does that mean?”. I still remember him explaining monotonicity to me a few years ago with the patience of a saint (St. Numbers).

The FOSDEM talk’s shaping up rapidly. I’ve now updated its title and description to better reflect its contents: “Willow - protocols for an uncertain future”. This is going to be a very different take on Willow than we’ve presented before, so I hope it will be interesting even if you’ve been hanging around the project for a while.

This week, I want to start adding more random little one-off drawings to our weekly updates.

~sammy

I have been programming too much over the past weeks; it has been impoverishing my life. Case in point: tomorrow (I am writing this on Wednesday), I am leaving for p2p-basel, and I... have barely thought about that at all, being too busy doing computer things. I don't even have a presentation to give, whereas in the previous years my problem usually was having too many presentation ideas to choose from. Still, there is much to look forward to: I will be seeing old friends again, will meet new interesting folks (there'll be someone from the Pijul VCS, I might finally get closure on some question regarding this paper), there will be interesting discussions and fun evenings. And I'll be meeting Sammy in person for the first time in over a year!

Despite me programming too much, we will not get everything ready by FOSDEM that we had hoped to. Which makes me feel disappointed, despite the fact that I have actually been pretty productive: Bab slice verification is implemented, and I am close to having a simple storage backend that can store a single, contiguous slice. In terms of unlocking funding milestones, I am more productive than needed. Yet the self-imposed FOSDEM-deadline makes me feel bad about the current state of things.

Hence, I label the self-imposed deadline silly, and I will do my best to focus on everything I did implement instead of what is still missing. As usual, there is also the voice that wants Willow to finally become a real thing that people use. But if listening to that voice means wasting my time staring at a small rectangle all day, then I label that voice silly for now as well.

In less silly news, academia and macadamia have a Levenshtein distance of two. Hmmmm, macadamia... now that is something I can get behind!

~Aljoscha

This week my attention was once again fully occupied by the upcoming FOSDEM talk. After a run-through of a draft slide deck this morning, I felt dissatisfied, although everything was arguably there. Empty calories.

I am a person driven by dissatisfaction, and I work on things until I am happy with them. Aljoscha has the same malady. There is seemingly no cure. This makes the things we work on take a very long time. This website is a kind of compromise: we blog about our ongoing dissatisfaction on a weekly schedule.

So now I am restructuring the presentation for what is hopefully the final time. I mentioned last week that if you want to learn about what Willow is and how it works, we have a website. But why is Willow like it is? What kind of world made us feel such a thing was necessary? That is the kind of talk I think people will enjoy most.

It will be fun, maybe even insightful. If you have been reading worm-blossom.org for any amount of time, how this presentation turns out will not surprise you in the least.

~sammy

(also, Bab implementation notes)

~Aljoscha