Migrate Orbit FS stores to flat-file databases, internalizing attachments as blobs

[andymatuschak]

Orbit's on-disk file format is presently a bundle: a folder containing a SQLite database and a folder of attachments files. Flat-file formats are so much easier to deal with, though. Want to copy or back up your database? Just one file. Likewise, serving bundles over HTTP is a pain. And you've gotta keep your database and your associated files in sync. Most apps that go the bundle route usually end up developing a parallel .zip-based file format (in fact, Anki has done so!)

I'm persuaded by the arguments in SQLite As An Application File Format that we'd probably be better off just moving the attachment blobs into SQLite and using a single flat file as the data format. In many cases, this will actually be faster to boot!

[andymatuschak]

Unfortunately, React Native makes this difficult. None of the SQLite native modules support BLOBs. And in fact, the (traditional) Native Modules bridge doesn't support marshaling binary data at all, so we'd have to move to a library which uses the newer JSI bridge, like https://github.com/ospfranco/react-native-quick-sqlite.

I filed an issue about BLOB support: ospfranco/react-native-quick-sqlite#11. It'd probably be a rabbit hole, but we could probably add support ourselves if need be.

Still, all this makes me less enthusiastic about moving in this direction. More work than I want to spend right now.

[andymatuschak]

180-ing here: expo-sqlite is doctrinaire about where SQLite databases are located, and won't actually permit the "bundle"-style file format anyway. So I added BLOB support to react-native-quick-sqlite (which should be more performant anyway, since it's JSI and hence should be a much cheaper FFI). Working through the rest now…

[andymatuschak]

Alright, this is now implemented. .orbitStore is now a flat SQLite file.

@ZanderKasanjian, I think you're the only one using Orbit on your local machine, but I'll note this publicly in case any others will be affected. With these changes, you'll need to do a "full sync" of your Orbit account to your machine; I didn't bother implementing a migrator because there are so few local users. Please:

1. Before pulling the changes in 04d4ce0 and later, please sync to make sure all your local changes are saved on the server.
2. Delete your local .orbitStore folder (or move it aside)
3. git pull, yarn install, and yarn build in the root.
4. sync again to a fresh .orbitStore instance—this will now be a flat SQLite file.
5. Carry on as usual! note-sync should behave as before.

[ZanderKasanjian]

@andymatuschak Thanks for the alert! Followed the steps with no issues.