worm-blossom

At the workshop in Prague, a surprising number of people had been on Secure Scuttlebutt (SSB) at some point, and SSB also kept popping up as a reference point in conversations. And a few days ago, someone on our Discord was asking about the relation between Willow and SSB. So I'd like to reflect a bit on how SSB still influences my design work.

I still subscribe to several of the mantras of SSB. The most important ones:

• No global singletons: if a system depends on any kind of global singleton — be it a server, a network, a legal entity, or a blockchain — it is brittle and can will be co-opted.
• Against consensus: those parts of a system that depend on global consensus between all participants will be unable to evolve as needs change. My personal take on this mantra is more nuanced than its classic formulation: I believe that there is value in a minimal foundation that is guaranteed to remain unchanging. Eventually it will have to be replaced because of its inability to adapt. But until that point, it will provide a stable foundation, superior to a continuously moving target that accrues complexity over time. As long as such foundations are clearly labeled and allowed to retire eventually, things are great. But: these foundations should be minimal. And everywhere where evolution is desirable, there should be no need for global consensus.
• Free listening: free speech has to be counterbalanced by the ability not to listen. Listening should be opt-in, not opt-out.

Many other aspects of SSB have not aged well. I've detailed my views on immutability-by-default elsewhere. Another pattern I would not want to see repeated on top of Willow is the public follow-graph of SSB: everyone publicly recording whose data they want to follow. While this enables a nice replication mechanism with the ability to transitively fetch data for improved reliability and discovery, the public nature of this graph also enables spidering and surveillance. This tradeoff feels inacceptable to me by now.

More generally speaking, SSB was emphasising discoverability and interlinked data, in order to optimise for network effects and virality. The public follow graph, immutability, a single multi-purpose log per user... many SSB design decisions are in support of fostering network effects. But these are also the design decisions that make the protocol unsuitable for vulnerable users.

The tradeoff certainly payed off, chances are SSB would not have reached its level of adoption without these deliberate design choices. Perhaps projects such as Willow are fated to being crowded out by projects that emphasise virality over privacy. At least as long as the majority of tech users are not vulnerable users themselves. This feels sobering, but it cannot change the kind of technology we are building — because we desperately need this tech to be available and reliable.

So while the average techie is safe and confortable, I guess we'll simply have to rattle a tin can every once in a while.

~Aljoscha

A prominent topic of last week’s gathering in Prague were sneakernets. There is something about this method of transporting data — the ultimate fallback mode of peer-to-peer transport — that seems to capture people’s imaginations. Embodied control of where data is, and tacit recognition of whose hands it is in, understandable to anyone.

On the first day I found my own understanding (and implementation!) of sneakernets being interrogated. What is Willow’s view of the sneakernet? Oh, so a user needs to choose what is in this ‘drop’? How do user's express what they want from others? They can’t? I see.

By the next morning, my view of sneakernets was dismantled (alongside a few others) to construct a new view, one where sneakernets take on an almost homeostatic nature. In this model of the sneakernet, data is not only pushed into the world, but the need for certain data is also expressed. The overlap between these two forces creates a hot zone of data worth transporting.

Reeling a little from this model, I started wondering if there was a way to make it so that users could query a Willow drop, shifting its design from linear access to a random access (thus neatly nerd-sniping Aljoscha, who mentioned this idea first thing last week).

Then Aljoscha announced at breakfast that sneakernets were really about congestion control. As I understood it (while trying to carefully eat from a bowl I’d filled with entirely too much yoghurt and fruit), if you have a USB drive filled with data from many people, at some point you’re going to hit storage constraints and have to choose what’s worth sending. This is analogous to throttling, and is effectively congestion control.

But here’s the tricky (Aljoscha: fun) part: sneakernets are non-interactive, so you have the problem of congestion control without explicit feedback. But perhaps you could use implicit feedback of tracking which data you receive from other people, and use that to prioritise what to send yourself?

A day after the event ended, we received this message:

We are the Memory Commons, just that nobody told us about this when we met in Prague. It is memory that we care about. Some call it replicas, other fancy about USB sticks. When discussing Sneakernets, Aljoscha suggested that this is a discussion on congestion control, which now makes even more sense as congestion control is about memory of the router packet queues, managed by a social contract that can't be technically enforced. Memory commons all over the place.

When we take this view, we must be mindful of the distinction between a commons and a hoard. We spent the last thirty years building the memory hoard. This hoard’s purpose went far beyond archival, and into the wholesale weaponisation of data. While the constraints of peer-to-peer certainly encourage us to prune and forget data, we should still steward the memory commons mindful of the fact that the value of data is in how it affects us, and nothing more.

~sammy

I went to a workshop, and I was not immediately struck down by illness. Yay! I am completely exhausted as I am writing this on the train back: the workshop was well-facilitated and the participants were great, so I ended up overexerting myself quite a bit. I do want to share some notes I took during a fun session of discussing the limits of CRDTs.

Also, I may have gotten nerd-sniped into sketching an alternate version of the Willow Drop Format (with the creative working title “Super Drop Format”) that supports efficient seeking within encoded drops: a super drop file would store a three-dimensional search tree, so you would be able to obtain all entries in a given area in time logarithmic in the total number of entries in the whole drop, or linear in the number of entries in the area (whichever is worse). But I'll be pragmatic and give this format extremely low priority. Probably.

~Aljoscha

Like Aljoscha, I am completely spent after a very good week. It has been 24 hours since I got back home and I think I could still do with another big sleep. Was this because of all of the thought-provoking discussions we had, or the inevitable fallout of ingesting more sugar in four days than I have in the last two months? Who can say.

For that reason, I am going to shower you in drawings today, and go in depth in the coming weeks. I have so much material on sneakernets, designing new kinds of UIs for p2p systems, and thoughts on collaboration in our little niche space that we're good until May probably.

But, if the other attendees — or especially the organisers — of the event are reading this: thank you so much for a wonderful week.

~sammy

This week I did some admin things, and took care of some teaching obligations (as fun as worm-blossom is, I still have this actual job of mine). So not much to report on my side.

I did open some issues in the Bab repository, ranging from the pretty specific and contributor-friendly over more vague problems to the document-now-and-then-forget-about-it-for-a-long-while. Just writing a whole bunch of stuff down so I can focus on other things. Foremost among those other things being a test setup for our persistent store implementation.

With the workshop in Prague and then the Easter holidays, progress might be a bit slow over the next weeks. But we’ll see how it goes.

Finally, I thought a bit about what a stateless protocol for exchanging data between mutually untrusting peers could look like (the WTP is stateless and assumes a trusted peer, Willow Confidential Sync is stateful and does not assume a trusted peer; these thoughts were about exploring a different quadrant). And there seem to be some inherent limitations: our techniques for working with untrusted peers (while still keeping data safe from active eavesdroppers) rely on establishing a shared context between the two peers and later referencing that shared context. This is an inherently stateful business, so these techniques will not translate to a stateless protocol. Even when weakening “stateless” to “stateful client but stateless server”, I’m at a loss how to achieve this. My intuition is that this might actually be impossible. But that’s a problem for future Aljoscha, like so many other problems these days.

~Aljoscha

Last Saturday I was sitting on the sidelines of my kid’s kendo lesson, listening to the soothing sounds of screaming children hitting each other with sticks. Just a few months before, I’d been sitting here fretting about my then-upcoming FOSDEM talk, when suddenly the whole concept for the talk came spilling out. And as luck would have it, the same thing happened again this week, just as I was preparing a lightning talk for Dweb’s virtual meetup on “The Latest in P2P and Mesh Technologies”.

The final storyline: how data first came to be grouped together; how unscrupulous people figured out how to capitalise upon these groupings; how nearly all collections of data are now modelled on this extractive design; and Willow as a kind of anti-design to that.

Really, this loops back to last week’s thought of data as a byproduct, but having been elevated to the product. How do we devalue data? Make it illegible, disposable, impose a half-life on it through mutability and forgetfulness. (whispering in your ear): Willow

Okay I’ll stop. The approximately ~80 people attending the call seemed to enjoy it, though! I’m pretty sure people like the idea of Willow. It would be really nice if we could give people a chance to like using it.

Which is… going to happen? Sooner than ever? Aljoscha and I have a persistent store and drop format implementation within reach. We also now have a secret project. It is secret in the sense that you do not yet know what it is, but not so secret that I won’t tease its existence on my blog. Can I give any hints yet? Well… the image I chose for the super secret Signal group chat for it was this:

That’s cryptic enough, I think.

Oh, and this coming week Aljoscha and I will meet in Prague to collaborate with many of our peers in the P2P / networking space. Hopefully this time Aljoscha won’t immediately be struck down by illness and we might even get to hang out. So next week we’ll hopefully have some interesting stories to share.

~sammy

This Monday I took the train to Rotterdam to go speak at XPUB.

Before we started, Doriane (who had invited me after we’d met at FOSDEM) gave me a quick tour of the XPUB facilities, way up on the top floor of a building overlooking the Nieuwe Maas. I was really enamoured with the atmosphere of their space, which had zine-filled shelves, boards crowded with posters, and students working away in cosy, personal spaces. It didn’t at all have the feeling of a curated display case, but rather a living, breathing space with work flowing through it. Nice.

And as for my part: we neatly avoided a sammy slideshow situation (thank you doriane) and instead had a kind of guided discussion for two hours, in which we hardly talked about Willow. It was great. I had a big chalkboard behind me on which we assembled a timeline of nearly three decades of peer-to-peer trials and tribulations, from Napster’s legal dismantling to the state-sanctioned deactivation of Apple’s Airdrop. Interjections from the rest of the room were frequent and always thoughtful, and when we finished at 19:30 it felt like we could have kept on going.

Like I said last week, I couldn’t leave it on a downcast note. After all, what are we going to all this trouble for? I argued that we had an opportunity to remember that data is just a byproduct, and that the point is for it to change us somehow. Peer-to-peer protocols have some unique features which encourage us to learn to become conscious stewards — rather than hoarders — of data.

Anyway. Great time, great people, hope to be invited back again some day.

But that’s not all I got up to this week. I also implemented payload storage in the new Willow persistent store, using the Bab implementation Aljoscha has been working on. Streaming, verified storage! It’s a beautiful thing. Or it will be a beautiful thing once we’ve got all the bugs ironed out.

~sammy

Do I write a thoughtful editorial, or do I simply finalise this rather late update? While I have no idea about the identity of the BOTTLENECK in Blossoquest, I sure know that for this week’s update I was the bottleneck. But maybe there is time for sharing a few fun ideas still.

1. A live-coding environment for shaping the sound of digital instruments. Basically, you edit a function that takes as inputs a frequency, a volume level, the time since the note was triggered, and the time since it was released, and you return a value between -1 and 1. This function is called 44000 times per second to produce sound. Oh, and the function has access to the samples it produced for the past x seconds, so that you can employ feedback and self-reference in shaping the sound. (shoutout to the best repo on github, but this would have a live-refreshable js interface instead of compiling rust programs into command line utilities)
2. A theremin-like, mouse-based (or touch-pen!) UI for playing the live-coded instruments. Should have a toggle for switching between continuous frequencies and a mode that discretises into the 12 notes of well-tempered tuning. And while you move the right hand on the frequency-volume domain, the left hand can do key-presses: one key functions as a sustain pedal, another represents whether to produce sounds at all (pressing and holding that key is like pressing and holding a note on a piano).
3. A language for writing down instructions for triggering the digital instuments. Basically, what happens if you take the good parts of Lilypond notation and free it from the complexities of score engraving. You end up with a language for structuring events in a flow of time. The language would “compile” into a stream of MIDI or OSC signals. And, the theremin-like UI has a recording function that saves live input in the form of that written language.
4. Also, for something completely different: cellular automata (think game of life), but with a notion of continuous force fields. Cells can emit a force field, and the values of the field instantaneously propagates between cell updates, with a configurable decay based on distance to the emitter(s). Imagine a tree-drawing automaton where each cell of tree trunk acts as an emitter on the force field of treeness, and when picking a direction for growth, cells of low treeness are more likely to be picked.
5. And how do I turn the oscillations of a cellular automaton into music?

ALSO: I have taken some work-in-progress drafts of other ideas of mine and moved them from obscure branches into a work-in-progress directory here at the deployment of worm-blossom.org. There is an incomplete but already informative spec draft of the media file formats I wrote about some weeks ago. There is a sketch of a bit-level variable length integer encoding for usage in the media formats. And there’s a draft of the website for Macromania. Without styling, and without most of the content, but the getting-started guide and the introduction-to-macros writeup are functional. You could start using Macromania today!

Oops, I suppose I did end up writing a full editorial.

~Aljoscha

I’ve had repetition on my mind this week, most explicitly triggered by Lu Wilson’s writing on the subject, but also earlier through some reflection on the music I’ve been making. My commentary for last week’s background music track was simply I feel like I'm repeating myself too much with these =/. By now, I’m much more willing to embrace the repetition.

Later, I went to a free improvisation concert played by Fred Frith and Liz Albee, and the weakest passages by far were those where they did not feel comfortable to simply repeat themselves.

And later still, a book I’m reading had the following A. N. Whitehead quote:

[..] no rhythm can be a mere pattern; for the rhythmic quality depends equally upon the differences involved in each exhibition of the pattern. [..] A mere recurrence kills rhythm as surely as does a mere confusion of differences.

I don’t think this makes a lot of sense, at least not when applied to music. No sound happens in isolation, everything is shaped by the sounds that came before. Even when repeating a pattern verbatim, the effect on the listener does not repeat: the first time you hear it, it is new. The second time you hear it, the impression drastically changes, because it is not new. The third time, it is a repetition of something that was already familiar. And so on.

Each repetition has a different quality, because it was preceded by a different context. The quality of change arguably decreases (the difference between the first and the second repetition is more pronounced than that between the 50st and the 51st repetition), but it keeps changing. After 400 repetitions you get a completely different effect than after 32 repetitions. Just like you cannot step into the same river twice, you cannot truly repeat the same pattern.

Oh, wait, this is supposed to be a dev diary, right?

Our Bab implementation now has a well-documented public API, and Sammy has started coding against it! I will spend the upcoming week writing more thorough tests, and next Friday I think we will release it properly. Writing tests is well and good, but I need to accept that I’ll release buggy libraries anyway. So might as well put a disclaimer on the release and ask for bug reports.

~Aljoscha

Early this week I made a few wireframes for a kind of generic Willow management UI (with Penpot, which is nice).

Discord’s ‘server’ UIs have always mapped very cleanly to Willow’s namespaces, and it seems negligent not to appropriate their work. But as soon as you start wading into the more unfamiliar concepts like keypair and capabilities the difficulty begins to reveal itself.

One of the most beautiful ideas about systems like Willow is using a single namespace as a bucket for all kinds of data. What if your Discord server also stored a wiki, a webpage, or anything else its users wanted to put into it? What if you could easily inline your wiki pages into your chats, and weave your data together? And had decentralised permissions which could stay on top of this menagerie of data?

Do you interface with that data in a single mega interface, or many seperate ones? If the latter, does each interface manage its own personas and passes (keypairs and capabilities)? In that case, do you reveal those nitty gritty details like being granted access to /chat/general or something aware of its context, like “General discussion”?

We've bonked our heads against this many times over the years. We tried a reusable React component that you could drop into your (presumably React-powered) app. We tried a kind of web-based mini-OS with different interfaces running inside a managed UI frame. And now I am just starting to think that the solution is a set of agreed conventions, spanning from representation to UI. I uttered the cursed phrase “cultural API” earlier, but I suspect this might just be a rubbish way to say pattern language.

Next week I’ll make an appearance at XPUB (a two-year master in experimental publishing in Rotterdam) to talk about p2p, Willow, and how we communicate our work. I’m really looking forward to this, and I’m challenging myself to find different ways to tell a story about peer-to-peer and Willow. For this one, it’s going to be a timeline, thirty years of lessons learnt the hard way. I feel like it would be irresponsible to only conclude we need to march on with grim hypervigilance. Surely we’re guarding something precious with that vigilance? It’s not just our safety, but something precious, joyful, and unrealised?

~sammy

Let’s start with nuts and bolts dev diary stuff: I started working on a new persistent store for Willow. It’s a straightforward port of the previous persistent store, which used sled under the hood. Except things were just different enough that I could justify switching to Fjall instead. Translating from sled’s APIs to fjall’s was pretty straightforward! This store is missing a few pieces still, as the Store trait needs to be updated with a few methods for storing and retrieving payloads, which depends on the Bab work Aljoscha has been feverishly boshing away at. So: we’re getting there!

While that bubbles away, what else to do but learn about Tauri, and seeing what I can glue together with our Willow Rust APIs? I got about as far as creating one button before my head filled with question about how we’re going to present all of this to human beings. Like the keypairs which are used to sign entries, for instance. they’re just not anything like user accounts on something like Discord, or even Mastodon.

Since the earthstar days, we’ve referred to the set of features keypairs confer as an ‘identity’. But to me, identity smacks of some kind of vague proving humanity rubbish. But that’s kind of exactly the opposite of what we want Willow’s keypairs to do:

• they can be used to prove authorship
• but you can have a lot of them
• and they have as much or as little identifying information attached to them as you want.

So how about ‘persona’? Data can be verified to be written by a persona, you can have as few or as many as you like, and you can attach as much ‘identity’ information (e.g. display name, avatar) you want… or not. Miaourt suggested using little masks as the iconography for this, which I really love.

I'm not the only working on Willow GUIs right now, so expect more of this!

~sammy

So much has happened this week! This calls for bullet points!

• Good progress on the Bab implementation: we have well-documented public API for (persistent) single-slice-per-string storage with support for partial verification. Still requires better tests, but then it can be released. Before that, I’m creating a Bab-aware trait for Willow stores in the willow25 crate though, to unblock Sammy’s implementations of a persistent store and the Willow Drop Format.
• I attended a DWeb preparation event on Saturday. Was nice to run into many familiar faces, and to get to know some new ones. Went home with renewed motivation to get Willow out into the world.
• I met with Zelf to record an episode of her podcast. It will probably be a while until editing is finished and this episode is released, though.
• On Tuesday, a bunch of us p2p nerds ended up in a bar with Olivia Jack of Hydra fame. Hydra is a super cool environment for creating 2d graphics — basically a synthesizer for graphics. I may have spent too much time playing around with it afterwards. It is so easy to get something fun going. I also may have started reading the source code and begun sketching a WebGPU-based implementation. The chances for that ever getting anywhere near useable are close to zero, but at least I had a very fun day of rest.
• I have strong opinions on variable-length integer encodings. So I was happy when I read Brooklyn Zelenka’s new bijou64 spec, which is an interesting variation on my older VARU64 specification. Slightly more complicated, but also slightly more efficient, and arguably easier to detect non-nanonic encodings. An interesting read for the other three people who care about varints.
• This week I finished a super secret side-project that I will reveal... later :>

~Aljoscha

When last week's update went up, I was at the Internet Archive Europe HQ in Amsterdam ('HQ' makes it sound like a big shiny building. It's really a cute little house by a canal). Remember last week when I said I felt like a frog finding their special frog habitat? Well this was more frogs. Many people who I'd been looking forward to meeting, and (somewhat amazingly to me) a fair number who'd heard of Willow before. After a lot of conversation, a sensible amount of cheese, and maybe a little bit of schmoozing, I left feeling energised. For years I've been trying to find a local network of people interested in this niche field, and either I was too much of a shut-in (likely), or this community in the Netherlands has finally reached a kind of critical mass and level of interconnectedness with the broader network.

Meanwhile, I worked on willow_rs. Unfortunately (and kind of intriguingly) my work finally presented a dead end in how well Rust is able to accommodate Willow's generic and parametrised ambitions. Aljoscha has (hopefully) done a write up on this below (don't leave me hanging aljoscha). I've started working on an implementation of the Willow Drop Format (WIP pull request here), where I am practicing liberal use of the todo! macro to implement as much as possible without letting little things like not having that dependency yet weigh me down. This is like taking a breadth-first approach where instead I've always taken a depth-first approach to missing bits and pieces.

~sammy

What is going on in Blossoquest? Since when did it even have that name? Why are we suddenly in a Western? Does the crack in the wall have any meaning? How can Sammy claim dislike for Homestuck and unfamiliarity with Hussy’s work, and yet the third panel looks like Problem Sleuth and the name like a play on Bard Quest? What is going on?!

Anyway, apparently I am supposed to write about the limitations of the Rust type system instead of investigating these mysteries. So here we go!

The Willow specs are highly generic. So naturally, our Rust implementation is full of generics as well. Because this is rather cumbersome, we are also providing a non-generic implementation of Willow’25 for application developers who shouldn’t need to care about the flexibility offered by the parameterisability of Willow.

In our previous codebase, the willow25 implementation merely provided type aliases, with the consequence that compiler messages bombard developers with all those hidden generics. So in the reboot of the codebase, we went for proper wrapper types. We even have a wrapper! macro that automatically generates the structs that wrap the underlying generic types. So far, so good.

The first problem that stems from this arrangement is that of converting between references. Suppose our generic implementation has a function that requires a &Entry<Lots, Of, Generics>. To implement the willow25 version of that function by calling the original function, we need to convert &Willow25Entry into a &Entry<Lots, Of, Generics>. Reference-to-reference conversions are not exactly the most idiomatic Rust. But this was doable, with the `wrapper!` macro even completely hiding the required unsafe and #[repr(transparent)] shenanigans that make it work. And up until Sammy’s recent implementation work, this was good enough.

The real trouble started however once we needed willow25 wrappers for generic types made up from other generic types. For example, a generic AuthorisedEntry struct consists of a generic Entry struct and a generic AuthorisationToken struct. A Willow25 wrapper around AuthorisedEntry would internally still use generic components. But for some code, we would need it to consist of (or at least to be interpreted as consisting of) a non-generic Willow25Entry and a non-generic Willow25AuthorisationToken. But for this interpret-the-generic-components-as-being-non-generic-wrapper-types business, the generics must be instantiated with types that satisfy certain constraints regarding their memory layouts. The exact details are gnarly and do not really matter, the important part is that we ended up in a situation where there would be unsafe code galore. This in itself is bad but not a showstopper, but the real problem was that the safety of this unsafe code would depend on whether you instantiated the generics in a certain way.

The compiler has no way of enforcing any invariants here. And I have never seen any code whose safety depended on properly instantiating type parameters. Calling this situation unidiomatic would not do it justice, I’d rather reach for an unholy abomination whose creators should be barred from writing Rust code ever again.

So what now? We will keep the generic code base for now, but new work will only provide willow25 APIs. Where possible, those will still make use of the underlying generic implementation, but where that does not work we will simply duplicate the generic code and then rip out the type parameters. Long-term, we will probably phase out the generic code base completely, doing concrete willow25-based implementations directly instead.

This is a bit of a shame. The genericity of the specs is a decision I still stand by, and generic implementations do have a certain elegance. But it turns out that (idiomatic) Rust is just not particularly well-suited to it. And we want to keep the codebase accessible and high-quality. The decision to move to non-generic implementations for now stings, but everything else would lead to an idiosyncratic code base that would have made Dominic Tarr proud, and scared off everyone else.

(Also, writing docs for two almost identical versions of our APIs was a pain from the very start, so I cannot truthfully report that I am entirely unhappy.)

~Aljoscha

Recently I’ve felt like a weird little frog who is finally finding their weird little microhabitat. The perfect temperature! Exactly the kind of slime I like! Other weird frogs like me! January put me in the path of lots of people who do and make things I’m excited about, and who seem excited about the work I’m doing too.

The only problem being that I actually have to do that work.

Our immediate goal with Willow is to implement the means to save and retrieve Willow data, and to have at least one way to exchange that data. For the latter, we’re going with the Willow Drop Format, which collects lots of Willow data into a single file that you can deliver with whichever way you like to send files.

Since I started working on private encodings last week, we’ve merged implementations of private path encoding, private area encoding, and private authorisation token encoding to willow_rs’s main branch. I had to get all these past the relentless scrutiny of our fuzz testing apparatus (and Aljoscha), so I’m fairly confident these new implementations are robust and well documented. Which feels great! There is definitely some kind of maladjusted satisfaction I get from merging code, which is something I should probably watch out for.

With that out the way I can get back to the important work of being a weird little frog. This weekend I’ll be heading to the Internet Archive’s new European Headquarters in Amsterdam for a little borrel, where I guess I’ll do borrel-y things like eat little cubes of cheese... and with any luck, meet some more frogs.

~sammy

I finally made good progress on the persistent backend of the Bab implementation. All the logic is in place; the remaining steps are to rearrange the public API, document it, and then to add more thorough tests.

And instead of writing a long editorial, I’ll continue programming now.

~Aljoscha