https://news.ycombinator.com/item?id=24337244

I watched the talk “Platform as a Reflection of Values: Joyent, Node.js and beyond” by Bryan Cantrill, and I think he was able to put into words something I already felt for some time: if there’s no piece of software out there that reflects your values, it’s time for you to build that software1.

I kind of agree with what he said, because this is already happening to me. I long for a database with a certain set of values, and for a few years I was just waiting for someone to finally write it. After watching his talk, Bryan is saying to me: “time to stop waiting, and start writing it yourself”.

So let me try to give an overview of such database, and go over it’s values.

I want a database that allows me to create decentralized client-side applications that can sync data.

The best one-line description I can give right now is:

It’s sort of like PouchDB, Git, Datomic, SQLite and Mentat.

A more descriptive version could be:

An embedded, immutable, syncable relational database.

Let’s go over what I mean by each of those aspects one by one.

I think the server-side database landscape is diverse and mature enough for my needs (even though I end up choosing SQLite most of the time), and what I’m after is a database to be embedded on client-side applications itself, be it desktop, browser, mobile, etc.

The purpose of such database is not to keep some local cache of data in case of lost connectivity: we have good solutions for that already. It should serve as the source of truth, and allow the application to work on top of it.

SQLite is a great example of that: it is a very powerful relational database that runs almost anywhere. What I miss from it that SQLite doesn’t provide is the ability to run it on the browser: even though you could compile it to WebAssembly, it assumes a POSIX filesystem that would have to be emulated.

PouchDB is another great example: it’s a full reimplementation of CouchDB that targets JavaScript environments, mainly the browser and Node.js. However I want a tool that can be deployed anywhere, and not limit it’s applications to places that already have a JavaScript runtime environment, or force the developer to bundle a JavaScript runtime environment with their application. This is true for GTK+ applications, command line programs, Android apps, etc.

Mentat was an interesting project, but it’s reliance on SQLite makes it inherit most of the downsides (and benefits too) of SQLite itself.

Having such a requirement imposes a different approach to storage: we have to decouple the knowledge about the intricacies of storage from the usage of storage itself, so that a module (say query processing) can access storage through an API without needing to know about it’s implementation. This allows the database to target a POSIX filesystems storage API and an IndexedDB storage API, and make the rest of the code agnostic about storage. PouchDB has such mechanism (called adapters) and Datomic has them too (called storage services).

This would allow the database to adapt to where it is embedded: when targeting the browser the IndexedDB storage API would provide the persistence layer that the database requires, and similarly the POSIX filesystem storage API would provide the persistence layer when targeting POSIX systems (like desktops, mobile, etc.).

But there’s also an extra restriction that comes from by being embedded: it needs to provide and embeddable artifact, most likely a binary library object that exposes a C compatible FFI, similar to how SQLite does. Bundling a full runtime environment is possible, but doesn’t make it a compelling solution for embedding. This rules out most languages, and leaves us with C, Rust, Zig, and similar options that can target POSIX systems and WebAssembly.