Alternate solution for sync IDB calls in Safari?

[dfahlander]

Hi!

Congrats for this project! It has the potential of a game changer. I was actually having the same idea after reading hosting-sqlite-databases-on-github-pages earlier this summer. But ideas are one thing - creating something that works with all the details and deep problem solving that you have made with locks and everything is impressing 🥇

My thoughts of solving the async problem was to utilize synchronous XMLHttpRequest and intercept the request locally from a service worker and do the async indexeddb jobs. I suppose this might be heavier than your solution with SharedArrayBuffer and Atomics, but maybe it could be an option for Safari? Thoughts?

[jlongster]

Hey, thank you!

Unfortunately, I don't think you can intercept synchronous HTTP requests in a service worker. You can check it out though and see if I'm wrong. But the good news is SharedArrayBuffer/Atomics will happen in Safari, so the current fallback mode is just a short-term solution. I wouldn't worry too much about the lack of SharedArrayBuffer because all browsers will have it eventually.

You might care about supporting older browsers though (which is why we have a fallback mode). The sync request would be very annoying to implement, however, and the more I think about it, it won't work. The internal worker that performs IDB requests also needs to block. The nice thing about Atomics is that both sides have the ability to block. Since the internal worker can block, it can keep an IDB transaction alive for as long as it wants. Without that capability, you'd lose out on major perf improvements. You also lose out on robustness; we have to keep a write transaction open for the duration that SQLite has the file locked. That's the only way we can ensure that all writes will be written in the order the SQLite expects (essentially: we map locking semantics onto IDB transactions).

So unfortunately, I don't think it will work. But it shouldn't be too long before they are available in all browsers; it's just a matter of time. Until then, the fallback mode works well, the main limitation being that one tab holds an exclusive lock on the db permanently (so multiple tabs can't write).

[quolpr]

I am not sure, but maybe it can help https://blog.persistent.info/2021/08/worker-loop.html. I think the idea is similar to what @dfahlander described. But instead, importScript is intercepted(in ServiceWorker), and importScript makes sync requests

[dfahlander]

I'm hoping Safari would fix the SharedArrayBuffer support. But in case it would take years before it happens, I worry a bit over the current fallback solution:

1. it needs to read entire DB to memory
2. it fails if multiple tabs writes to it - and the other tabs won't see updated data (?)
3. it gets stale after an error.

(correct me if I'm wrong ?)

There is a trick to keep an IDB transaction alive asynchronically - which could solve the locking need for the inner worker. We provide it for some edge cases in Dexie (like waiting for the async methods from webCrypto to complete while keeping the transaction alive) Dexie.waitFor(). The idea is described in this original idea. It's definitely not a nice solution either - but it works and is asynchronic so that the worker could wait for stop-message and keep transaction alive until it gets it.

Well, the net cost of all this (XMLHttpRequest / importScripts() + service worker + tricking transactions to be alive asynchronically + all the complexity of it all) may not be worth it of course. Especially if SharedArrayBuffer for Safari is around the corner(!) Just in case it would take to long for it to happen.

In the end, I hope we will be seeing a new W3C standardized low-level file system API worth its name in the future (OS-level file access, in a sandbox of course) that could be used instead of something that sits on top of another DB engine. Wouldn't surprise me if that happens some day, let's hope :)

[jlongster]

Yeah, someone on twitter pointed me to that post. Brilliant use of importScripts because service workers are not able to intercept sync http requests.

I still don't believe it will be as reliable as we need it to be, given that the internal worker can't block as well.

@dfahlander That solution basically just infinitely keeps the transaction alive by doing work. I highly, highly doubt that will give us the performance increase of keeping a transaction alive. Everything is so sensitive here and even if it just does 1 extra db call interleaved between all the reads/writes, that's going to kill performance.

Well, the net cost of all this (XMLHttpRequest / importScripts() + service worker + tricking transactions to be alive asynchronically + all the complexity of it all) may not be worth it of course

Yes I agree. Feel free to try the hack, but not only is it more complex, I don't think you'll see the same levels of performance. It probably be 3-4x slower, which sometimes still beats using IDB raw but not by much, and in general is too slow for heavy usage. (The 3-4x number comes from experience before implementing certain optimizations like keeping transactions alive)

(If anyone does this, it should be a new backend instead of being inside indexeddb because there's already enough stuff in there)

Yes, the fallback solutions has drawbacks for sure. But it's fine if you just want the app to run in one tab. I bet that's fine for most use cases; things that are going to use this are complex apps that people probably open once and use for a while. Also, Safari has a very small market share. Of course, I wouldn't be satisfied at all with that long-term.

You can already run Safari with SharedArrayBuffer/Atomics with a flag (don't have it on me right now). They recently added that flag, and there's good indication that it's going to happen soon. I have nothing to back this up, but I wouldn't be surprised if it was enabled within the next year.

[jlongster]

In the end, I hope we will be seeing a new W3C standardized low-level file system API worth its name in the future (OS-level file access, in a sandbox of course) that could be used instead of something that sits on top of another DB engine. Wouldn't surprise me if that happens some day, let's hope :)

There is one already happening! Google is very interesting in solving this problem, so there's a chance we'll get it. I'm pessimistic, given how much these proposals have to go through to get standardized, and that it will 100% meet all the requirements, but here's hoping.

The latest one is here: https://github.com/WICG/file-system-access/blob/main/AccessHandle.md

I am discussing this proposal with the authors, and hoping that we can use absurd-sql as a good test case to make sure it meets requirements