Last week, I embarked upon an experiment: could sneakerweb be made to work with Rocket, the multi-threaded web framework? The good news is that Ufotofu now has a compatibility layer with tokio (see announcement)! The bad news is that the answer is no, sneakerweb cannot work with Rocket.

We can get ufotofu and Willow’25 working with tokio, but only with the current_thread (i.e. single-threaded) flavour of tokio. Rocket requires a multi-threaded environment.

Ufotofu and willow25's incompatibility with a multi-threaded environment is due to our adherence to Frugal Async Rust. I am going to admit to not having strong opinions on this. I do wonder if I should be concerned that Willow and Ufotofu can't work in multi-threaded runtimes, or whether I happened to stumble into the one situation (Rocket) where this was a showstopper.

Instead I will be using a web framework compatible with tokio's single-threaded mode, probably Axum. The motivation remains using someone else's HTTP request handling and support for templating, something which we're kind of winging ourselves right now.

Otherwise I am doing some up-front work for next week's update.

~sammy

Another scattered week for me, rang in by a lovely choir reunion and then dominated by the cold incurred therein.

I did make some more progress on the WTP spec, but it still isn’t in a shape where I can meaningfully share it. I’m getting there.

Also, we held a call today with Olivia of wobbleweb, brainstorming ways forward for integration between wobbleweb and sneakerweb. As we were pondering the aesthetics of the Fediverse compared to those of myspace pages, Olivia coined this week’s motto of aesthetic sovereignty that immediately resonated with all of us.

And finally, I spent the worst day of the cold working on Macromania, porting in a half-daze several macro packages from the old version of macromania to the current version. Specifically, the new macromania-web package combines and improves upon three separate old packages:

• There are macros for creating URL references, now supporting references to files living on different domains.
• There are macros for moving assets (stylesheets, scripts, etc) in a web project from an input to an output directory, and for applying transformations (compilation, minification, etc) to the assets, now also supporting dynamically creating assets without any corresponding input file.
• There are macros for automatically adding stylesheets and scripts dependencies to html files as they are registered by other macros.

Is this all overly complicated, niche, and of utter insignificance compared to progressing Willow? Probably, but on the other hand: $x^2\text{.}$ We can finally render KaTeX on this site. In a principled way. We simply type <M>x^2</M>, and it automatically does the server-side rendering and embeds the necessary stylesheet on any page where we use this. And we did not even have to manually place the stylesheets (and fonts) anywhere. This may have required several days of coding (and years of idle designing), but if this isn’t the worm-blossom way, I don’t know what is. And Sammy agrees, I think.

(more seriously, this lays the foundations for several nontrivial website features that improve how we can communicate our ideas, and it is hard to overstate how important communication is to all we are trying to achieve)

~Aljoscha

A little written field recording: the sound of my son sniffing behind me as I work, the chirps and beeps of a Popular Monster Collecting Game. School's back in and so are the colds.

One of the things I did this week was write the new Store tutorial. The workflow for these has solidified (and uh, expanded) since we first started publishing these. A month ago I pushed up a willow-tutorials repo with all of the tutorials in a single workspace. This repo acts as a space for drafting the tutorial code and also ensuring the things actually run. Then I generate a JSON dump of the willow25 APIs using rustdoc, and commit that to the willowprotocol.org repo, which has a macro for turning rustdoc output into definitions we can refer to on different pages (so that I can turn a reference of rs-willow25-storage-Store into a link that points directly to its corresponding documentation). Then I write up the steps of the tutorial itself, with code samples and output! It's a lot of rigamarole but I think these things come out pretty nice.

I've also started my work on support nodes for the new Pillow project. These support nodes are going to be configurable Willow peers which can communicate with any other peer which understands WTP. You'll be able to configure them with capabilities (so that they know what to sync), set storage limits (so that your Pi with 4gb of storage can pitch in), and configure if it should sync with other support nodes too. So it's got to be a little TCP server which can field WTP requests, and a HTTP server which serves up a web UI for administration.

In true 'I'm doing research for my project!!!' fashion, I've been looking at different Rust HTTP frameworks, and how they handle things like routing, validation, templating, etc. I quite like the look of Rocket, and I can easily imagine how to structure the routes I have planned using it. But I'd like to get some practice in with it before that. So here's my idea: I think I'm going to refactor sneakerweb to use Rocket for its web server. This will be very useful as sneakerweb's current HTTP server is single-threaded, and handles requests at a very relaxed pace, while Rocket's is multi-threaded. It will also make it much easier for us to add new pages to the sneakerweb server, and we'd like to make it possible to do pretty much everything you can through the CLI via a web UI as well. And with that, I'll feel confident enough using Rocket to use it for the Pillow project.

Speaking of sneakerweb, I have a lot of sneaky in-person events on the horizon, and this is lighting a fire under me to update sammy's guide to the sneakerweb, which to date has only been distributed in person. The first edition was a single page, the second edition has many pages and is set to be a real guide which teaches you how to do everything. Maybe it'll make sense to make a .snk containing this site available on the world wide web? I will confer with my colleagues.

~sammy

This week I began work on the new WTP specificiation. That’s it, nothing online to show for it yet.

But also, I had some thoughts that got me more excited to work on some important Macromania macros again. And those thoughts in turn are related to the sneakerweb. How so? About two years ago I was thinking a lot about scientific publishing, both in terms of the underlying incentive structures and logistics (they suck) and about the purely technical level of how PDF-based knowledge exchange has its drawbacks. I wrote the Cardboard specification as an alternative for PDFs: instead papers would be written as static websites, and bundled in archives that would allow shipping all (transitively) cited papers together with the paper itself. But then I never implemented software support for this specification or wrote any actual Cardboards. (The G-tree paper was a fruitful exploration of web-based scientific writing though.)

Last Saturday, out of nowhere, I remembered the Cardboard spec, and realised that the sneakerweb enables pretty much the same thing. And we already have running software. The minimal additional featureset I’d envision is a way for sneakerwebsites to specify other sneakerwebsites they depend on (i.e., link to). Perhaps as a simple /sneakerdeps.json file storing a json array of sneakerweb domains. This file would enable two key features: first, it would be possible to take a sneakerweb collection and extract from that everything required to fully and transitively browse a single starting website (or set of starting websites). And second, it would be possible to implement partial downloads of large “papers”: instead of downloading a .snk file containing all transitive dependencies, you’d download a .snk file containing only the starting website, and then you could recursively download its dependencies on demand (and whether you’d download eagerly or lazily, you could skip sneakerwebsites you already had available locally).

The sneakerdep mechanism could be useful for a whole bunch of things beyond scientific writing, in fact all sneakerwebsites that link to other sneakerwebsites would benefit from it. You could have nice software support where user agents could issue helpful warnings: both when a link to a dependency cannot be resolved because that dependency is unavailable, and when linking to a sneakerwebsite that is not listed as a dependency yet (to help authors keep their depencies up to date).

~Aljoscha

We're resilient creatures that can adapt to a surprising amount of stress. Before reaching a breaking point, there are several other intermediate points where things start to get a bit wobbly and plans start to go awry.

As this is a blog about making things and about being a person who makes things, it seems negligent not to mention that I hit my wobbly point a while ago, and that it's been affecting my ability to make things.

Would you believe me, reader, if I told you that updating this website each week started to fill me with apprehension? This cute little blog where no-one is making the rules apart from myself (and some might argue, Aljoscha)? Which we update without fail every Friday for nearly a year?

Regardless of our stringent publishing schedule, I recognise that my apprehension is not really about worm-blossom.org. It's about me, the rapidly changing circumstances which have brought me to my wobbly point, and recognising that if I don't want to hit my breaking point I need to do something about it.

So I'll briefly mention what nourishes me right now — maybe the only thing that really nourishes me right now — which is being in the physical presence of other people who care. About what they're doing there, about each other. When I'm sitting in that I feel like I'm part of a circuit, that there's nothing more precious than to be part of that interchange.

~sammy

Between the final days of my vacation and then catching up with work (the semester is starting soon, so there's a flurry of admin stuff to prep for teaching), I had barely any time for worm-blossom. So nothing to report this week.

Instead, have a random programming language design sketch! Imagine a statically typed, object-oriented programming language where all method calls must be mediated through an interface. Classes do not have ad-hoc method implementations, they can only implement interfaces. And concrete classes cannot appear in type signatures, only interfaces can (with semantics analogous to those of using interfaces as types in Java, or to trait objects in Rust).

The impetus for this (slightly cumbersome) design is to combine the flexibility originally envisioned by OOP — every object can be transparently replaced by a different kind of object as long as the new object reacts to the same methods — with static typing. In a language like Java, you lose the fungibility of concrete types the moment you constrain a variable or argument to a specific class. This interface-only type system would enforce fungability everywhere.

This has some interesting properties: for example, an equals method would have to implement comparisons based on public trait methods rather than by accessing internals. This feels like a bit of a hassle, but also it makes sense philosophically: equality (or equivalence, more generally) must depend only on publicly observable properties rather than inaccessible state. A bit like whether a person is perceived as a good person depends only on their observable actions, not on the belief system sustaining their actions. This type system design applies that viewpoint consistently to all programming.

Generics play a fairly important role in this type system: compare the kinds of values that could be returned by newer(a: Entry, b: Entry) -> Entry with those that could be returned by newer<E: Entry>(a: E, b: E) -> E.

Finally, this language should probably have some unconventional type-level operators: A && B could denote the set of types implementing both the A and the B interface. And A || B could denote the set of types implementing at least one of the A and B interfaces. On the value level, there'd then be a match statement for different branches depending on whether a value of type A || B implements A or B.

But enough of that, I have emails to write and access rights to assign =S.

~Aljoscha

Okay, so hear me out. I was listening to Kate Bush's Deeper Understanding while out on a walk. This is a song about someone who develops an unhealthy relationship with their computer via a piece of software which is able to talk back to them. The original version of this song was released in 1989.

The chat interface has now been around for decades. People have built new friendships, fallen in love, and built entire communities through speech bubbles filled with text.

And without that priming, without the education that on the other side of that speech bubble there is a living, breathing person, would any normal person have given LLMs a second look? Or would they have seen how absurd the premise is immediately?

This is but one of the many half baked thoughts I bothered others with after attending BYOW - A Webcraft Meetup. In my corner of the world it feels like I know of a lot of interesting, caring, talented people who for whatever reason never seem to get together. But we're all atomised away from each other. Why is that? The country I live in is dense and small. Is this the dark side of the bicycle, that anything further than 15 minutes away is just too far? (please keep tuning in for my broad and sweeping societal theories).

But back to the BYOW meetup. It was interesting that a lot of people were building shrines. Eulogies, making wishes, shrines to fandoms. There was always this big question of how the author went about the upkeep of the site, how pleasant or unpleasant that was and how they kept up with it anyway.

Speaking of which, while talking about this very website, I admitted that I still hadn't gotten today's update ready and was starting to feel nervous about it. And almost immediately help was offered. I've included a few contributions to this week's update from lovely people I met last night.

~sammy

This week I want to share some thoughts not about Willow but about the Sufficiently Good Media Formats, specifically about the problem of enabling efficient seeking to arbitrary timestamps in an audio or video file. As a bare minimum, you want to wrap the encoded samples in segments, say, of one second each. But how can we enable efficient access to specific segments?

The most straightforward way is to have a table of contents at the start of the file, specifying the location (i.e., the starting byte index) of all segments in sequence. The problem: with this scheme, you cannot emit a file in a single pass over the samples to encode. You have to first encode all samples and track the segment starts in memory, and then jump back to the space for the table of contents and fill it in. This requires O(n) memory, not good. You could also jump back and forth between appending a new segment and seeking to the table of content and adding a single item, but that's pretty inefficient (even when applying the obvious optimisation of batching things). And finally, the whole scheme precludes schemes that simply emit the bytes of the file as a single stream with no backwards-seeking (unless you do two separate passes, one for computing table of contents locations and one for actually appending samples).

A simple alternative would be to have the table of contents at the end of the file instead of the start. This makes it possible to emit everything in a single pass without seeking, but it also necessitates O(n) memory usage. So here are two alternative schemes, which I haven’t seen anywhere in the wild (though, admittedly, I haven’t thoroughly searched either — I’ve mostly had some fun trying to solve this from first principles). Both schemes enable finding arbitrary segments in logarithmic time, and allow for single-pass, O(1)-memory encoding.

The first scheme works by building a backwards skiplist over all segments: each segment specifies the location of the predecessor segment, and the location of another, possibly much older segment: the n-th segment specifies the location of the n-p-th segment, where p is the largest power of two that divides n and is strictly less than n. Finally, the very end of the file specifies the index of the start of the final segment.

This scheme lets you seek to any segment in logarithmic time: you seek to the end, obtain the start of the final segment, seek to there. Then you check whether the non-predecessor link overshoots your target segment. If it does, go to the predecessor segment, otherwise follow the non-predecessor link. Repeat until you find the target.

Producing these files requires O(log(n)) memory: you need to keep a table of the addresses of all segments whose index, in binary notation, consists of a number of ones followed by a number of zeros. Updating that table takes only O(1) amortised time, so the encoding process overall takes only O(n) time, the same as without any segment seeking scheme.

The other scheme is based on self-synchronising codes. First, you make sure that the contents of segments do not contain zero bytes (there are efficient techniques for that). Then, you prefix every segment with a zero byte, followed by its index.

Seeking works by guessing a likely location for the segment you are seeking for, and then scanning forward to the next zero byte and looking up the index of that segment. If you guessed wrong, you take a more refined guess and repeat. This guessing procedure can be simple binary search, it can try to estimate how long most segments are to compute a more likely location, or it can be a hybrid (that would/should combine the worst-case bounds of binary search with the far better average-case behaviour of computing indexes based on average segment length).

In the worst case, this scheme incurs a lot of overhead during seeking, spending time sequentially looking for the next zero byte. But on the other hand, non-adversarial files probably allow for faster seeking than the necessarily logarithmic approach of the skip-list scheme. And emitting these files adds no asymptotic overhead to the encoding process at all, which is quite neat.

I don’t yet know what to go with for the Sufficiently Good Media Formats, and I certainly will not make that decision before properly reviewing the state of the art. But instinctively, I do find the scheme based on self-synchronising codes very appealing...

~Aljoscha

This week I've been working on things that aren't quite ready to be picked apart by the ravenous audience of worm-blossom.org.

The new worm-blossom 'mega bar' is now in operation across the many worm-blossom franchises. We deployed these two weeks ago, but the styling for each site wasn't completely right yet. Now you can skip across the worm-blossom-verse with ease.

I began working on new tutorials for Willow, now that the Store and Willow Drop Format APIs have been published. Writing a tutorial has the unfortunate outcome of making you see the things you've designed from a beginner's viewpoint. New areas for improvement have been identified.

Meanwhile I have been held in the grip of the phrase “nothing special”. When is “nothing special”... well, special? It's when you see how something has been assembled, and that all the parts are in fact quite boring, and that you too could make nothing special yourself if you wanted to.

~sammy

I took all my pondering over the WTP design and cast it into a single specification collection of Rust types for the different message types. This is a fairly dense artefact compared to an actual spec writeup, but I hope I can gather some feedback (from Heni in particular :3) nevertheless.

Aside from a continuous stream of Willow test vector improvements, that’s mostly it. I really hope that the WTP design will stay mostly stable; while it is fun to work on these designs it is also somewhat stressful, and I look forward to fleshing things out without reexamining the foundations all the time.

I will be offline for ten days of vacation now (again, what a decadent summer), for yet another choir trip. I prepared background music and an editorial for next week already, but I won’t be as responsive on the Discord server during that time. Sorry @Freckles =D

~Aljoscha

Reader, it wouldn't be unreasonable to look at last weeks plans of ‘support peers’ and think: that’s just another name for a server. You couldn’t find enough peers, so you had to turn to an always-online machine somewhere. Cop-out.

Well: nuh-uh. In this editorial I will argue that we are doing something meaningfully different from traditional servers.

Servers are (hopefully) characterised by their high uptime, but they are not fungible. If a Mastodon instance’s server is down, we can’t point our mobile client at another server and expect to have the same functionality available to us.

Peers in a p2p network are fungible. We can connect to whatever we can and exchange as much data as possible. But a peer running on a mobile device has only very intermittent uptime. It may only be available when an app is in the foreground, which is not very often at all.

A non-fungible machine with low uptime is just… a crappy server, so that’s not a very interesting combination. But a fungible machine with high-er uptime feels like it’s relatively unexplored.

What kinds of infrastructure are possible when high-uptime peers are fungible? Sure, you get a ‘swarm’ of high-uptime peers that are more resilient to outages, but… how about something more intentional, like a relay of peers turning on and off as solar power becomes available to them?

Servers are also usually far more capable machines than the devices they serve: more processing power, more storage, more authority. But we’d expressly like to enable peers which can provide high uptime but which have other constraints such as low storage or processing power. If we can prioritise the storage of recent data, operators of constrained devices like this are still able to meaningfully pitch in.

And those high uptime peers can even sync with each other! I think it’s meaningfully different from ordinary servers, and I hope we can use this as a tool to change how we think about supporting infrastructure.

~sammy

A good chunk of my worm-blossom time this week went into the Willow test vector repository. Before this week, that repo held inputs and expected outputs for a few Willow encodings, some of them outdated and thus actively misleading. Now everything is up-to-date again. Additionally, I started adding test data for more than just encodings: there is now also a directory for data-model related functions, such as determining whether an entry should overwrite another, or computing the lexicographic successor of a path (a task less trivial than it initially sounds).

There has been a flurry of other tasks on my mind, to the point where I started to feel a bit overwhelmed. Writing them all down is probably going to help, so here we go:

• WTP design:
  • handling proofs of knowing the secret for a write capability,
  • unifying the different messages that all request entries as much as possible,
  • determining how/whether peers should pre-announce which features they implement,
  • mapping out which messages can be sent as responses to which other messages,
• the ed25519 ambiguity issue of willow25
• updating to the new major version of the ed25519 implementation we use in our Rust work,
• mildly panicking about quantum computers breaking ed25519 in the not-entirely-unreasonably-far future,
• planning Rust traits for Willow stores that allow aggregating metadata (for RBSR) and for splitting 3d-ranges into smaller ranges containing roughly equally many entries (again for RBSR),
• more media file format details (in particular, I’m still exploring different mechanisms for enabling efficient seeking in audio/video files),
• the Macromania website,
• some macromania macros I should really write sooner rather than later (macros for integrating and possibly transforming assets for websites, macros for referencing these assets),
• a data base file format for 3d Willow entry storage,
• the prerequisite 3d-zip-tree work I need for that,
• providing recommended ways of encrypting Willow data (and metadata) at rest as part of Willow25 (we already have generic designs, but no specific instantiation for Willow25 yet),
• API and implementation designs for this encryption work in Rust,
• figuring out how to make encrypted sneakerwebsites a thing (in a backwards-compatible way, ideally),
• work on my writeup of p2p design principles/lessons,
• plans for the first birthday of worm-blossom.org,
• idle thoughts around more intentional background music tracks in the next year,
• adding more Willow (and Bab) test vectors,
• planning some test vector improvements,
• sketching a design for a Willow-powered alternative to Email (that I will probably write more about in the future), and
• a whole bunch of encryption issues pertaining to that Email alternative.
• Bonus stuff:
  • sketching a type system for an object-oriented programming language where there are no concrete object types but only interface constraints,
  • sketching an engine for web-technology-powered text-based single-player RPGs, with beepbox integration for (reactive) soundtracks,
  • reflecting on Hannah Arendt’s On Revolution and how her thoughts on the origins and nature relate to technical protocol.

Jup, okay, in retrospect I can tell why I was feeling a bit stressed out this week.

~Aljoscha