Aljoscha Meyer & Sam Gwilym - Earthstar + Willow
P2P protocol and APIs for collaborative and social applications
Decentralised solutions
Can you introduce yourself and your project?
I’m Sam, representing the Earthstar project. We’re working on Willow, a family of protocols for synchronisable data stores. These stores can be used for many different kinds of applications: traditional data storage, social applications, or resilient data archival, as a few examples.
The Earthstar project was begun in 2020 by Cinnamon, whose work put user safety at the heart of everything. Cinnamon sadly passed away in 2022, but since then we’ve tried to carry the torch they’ve passed on to us.
There are two of us working on Willow:
- Aljoscha Meyer, pursuing a PhD at TU Berlin.
- and myself, Sam Gwilym. I’m a programmer and illustrator and I’ve been working on the Earthstar project since 2021.
What are the key issues you see with the state of the internet today?
The key issue we see is fragility. Most networked services are built in a tightly coupled way where a single component failure can bring the whole service down, and users regularly lose access to their data. And even if they can keep 100% uptime, these services are usually housed within data centres which depend upon prodigous (and growing) amounts of electricity and water to keep running.
Building networked software this way assumes that we will always have equal or greater access to the connectivity, natural resources, and supply chains we do today. And that may be a dangerous assumption to make, one which will leave us disconnected from each other in the wake of a natural disaster, geopolitical event, or simply the next unscheduled service outage.
Our view is that we need more technologies for antifragile networking.
Another issue we see is that we’re overserved by services which attempt to connect everyone, everywhere.
How does your project contribute to correcting some of those issues?
Devices using Willow can connect to each other and collaborate on data, just as we do today. The two big differences are that they can connect to each other directly, with no privileged intermediary infrastructure like a data centre; and that they can disconnect from the network indefinitely, yet still be able to read and write data. Whenever they reconnect, they’ll be caught back up as though they were connected all along.
You may have heard of something like this referred to as “local-first”, or offline-first, and we’re far from the first to develop this kind of system. But we think Willow brings some important advancements to this space.
For example: deletion. In a traditional network, you know all the places the data is stored, so you can just delete it. But when a network of independent, local-first data stores wants something gone, a marker for what needs to be deleted needs to propagate across the network. These markers (or ‘tombstones’) can never be fully removed, so they accumulate over time and leak potentially sensitive metadata, possibly hinting at what was deleted, by whom, and when. In Willow, we’ve introduced a new technique by which we can delete many items but leave behind a single marker, reducing the amount of storage used and metadata leaked.
Being able to connect to other devices directly also poses the risk of connecting to devices we don’t know or trust. But we have an apparent Catch 22: we want to know if that untrusted device is interested in the same data as us, but we also don’t want to reveal our interests to each other in case we have nothing in common. We have to prohibit the possibility of malicious actors collecting our interests, whether that knowledge is used for targeted marketing or political persecution. Willow uses a cryptographic technique called private set intersection which ensures that others can’t discover what you’re interested in unless they already know about it themselves. A device with nothing in common with yours is given the networking equivalent of being blanked.
Efficiency is really critical in this context. Not only because of all the extra communication we have to do in a system like this, but also because we want Willow to be able to run on very low-spec hardware. In order to do this, we have devices continuously communicate their memory constraints to each other, so they only ever send each other as much data as their partner can actually handle. Devices also tell each other what they’re interested in, and how much of it they want, so that e.g. a phone with very little storage space can always get the last 100 messages.
What do you like most about (working on) your project?
A lot of what we do is writing specifications for Willow, so communication is a huge aspect of our work. We’ve maybe over-indulged ourselves in thinking about ways to make Willow more comprehensible, and filled our website with illustrations, diagrams, and an interactive cross-referencing system.
We’re working in a space with a lot of precedent and many parallel projects. It’s interesting to see what lessons everyone has taken away from previous efforts, and a lot of fun to bring our own take to the table. I think we’ve now seen this space’s Cambrian explosion, and we’re now entering the Devonian era where all these slimy creatures are climbing onto land for the first time with their half-leg-half-flipper things.
Where will you take your project next?
Our long-term goal is to make Willow as boring as possible, as in ‘PVC piping under the sink’ boring. We want to move this technology to a place where application authors and users take these benefits for granted, and it's out of view, doing its job.
Our next step in this direction is to implement Willow in Rust, which will let us make good on the efficiencies we’ve been touting, and make Willow available on many platforms. This implementation work is being funded by NGI Core. Willow’s specification is more or less stable, so with any luck we can focus on continually locking down and optimising that implementation, and also let it act as a platform for academic research, e.g. for interesting data structures for Willow’s three-dimensional data-model.
I’m also excited to see where others will take Willow next, e.g. Iroh. They have a Willow implementation of their own in the works, but I hope we can eventually get everyone on board with a single implementation that serves the whole ecosystem.
How did NGI Assure help you reach your goals for your project?
NGI Assure funded the Earthstar project twice over two years, and having that sustained funding let us figure out which questions we should be asking, and even try to formulate some answers to those questions. This would have been impossible in a commercial context where we probably would have been pressured into pivoting into AI somehow. Conversely, an academic setting would not have been grounded enough for a project like this to develop into something people could use.
Do you have advice for people who are considering to apply for NGI funding?
Don’t over-commit and try to stuff your grant proposal with a hundred things in a bid to make it more attractive. Either do just enough to explore your idea and feel out its edges, or focus whole-heartedly on your core idea and delivering something solid and nothing else.
Do you have any recommendations to improve future NGI programmes or the wider NGI initiative?
Nothing that I don't think NGI hasn't been made aware of already, really: that we need more integrated, sustained projects, with different kinds of specialities involved if we are going to get to the point where there are viable alternatives to the proprietary services many of us rely on.
Acknowledgements
Images: Created by Sam Gwilym.
Published on September 16, 2024
Earthstar received funding through the NGI Assure Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 957073.
