Run absurd-sql in a serviceworker

[klehmann]

I did some quick tests to run absurd-sql in a serviceworker, but it fails like this:

serviceworker.js:63 Failed to open the database Yb.w.ErrnoError.w.ErrnoError {node: undefined, Oa: 20, message: 'FS error', hd: ƒ}Oa: 20hd: ƒ (c)message: "FS error"node: undefined[[Prototype]]: Error
at Object.Yb (webpack:https://absurd-example-project/./node_modules/@jlongster/sql.js/dist/sql-wasm.js?:160:231)
at Object.lc (webpack:https://absurd-example-project/./node_modules/@jlongster/sql.js/dist/sql-wasm.js?:160:402)
at eval (webpack:https://absurd-example-project/./node_modules/@jlongster/sql.js/dist/sql-wasm.js?:177:15)
at new Promise ()
at initSqlJs (webpack:https://absurd-example-project/./node_modules/@jlongster/sql.js/dist/sql-wasm.js?:24:24)
at openDB (webpack:https://absurd-example-project/./src/serviceworker.js?:19:77)
at eval (webpack:https://absurd-example-project/./src/serviceworker.js?:47:20)
eval @ serviceworker.js:63

Before diving deeper I would like to know if this is something that is supposed to work in general. My idea was to serve static HTML/JS files and images out of an absurd-sql instance and add some sync capabilities to fetch them from a server-side Sqlite DB.
But there's probably a workaround in using the Worker from the sample project and store duplicates of the web resources in the Cache Storage API.

[tantaman]

Is the idea that the service worker would intercept calls to your server and serve whatever it has locally in the db?

random observation on the particular use case -- I'm not sure sqlite is the best fit for storing images and static files.

[klehmann]

Yes, that’s the idea, to have a web application that is synced onto the device/desktop browser and can run offline. When online, app data and HTML/JS is synced with the server.

[mdubourg001]

After trying hard for a few hours, I finally managed to run absurd-sql in a ServiceWorker (for a NextJS app but not important here), here's how I did it:

1. initBackend wants a Worker instance in parameter. In the case of a ServiceWorker, the Worker object is navigator.serviceWorker.controller:

if (navigator.serviceWorker.controller) {
	const worker = navigator.serviceWorker.controller;
	initBackend(worker);
}

2. you need to tell Webpack not to resolve requires of native node modules (fs, path, crypto) (related to issue can't resolve fs #27):

// webpack.config.js

module.exports = {
	// ...
	resolve: {
		fallback: {
			crypto: false,
          	path: false,
          	fs: false,
		}
	}
}

3. absurd-sql needs to download a WASM binary in order to work (this is something that took me some time to figure out): this is not really documented but the binary can be found in the example repository src/examples/sql-wasm.wasm, and must be placed in order to be served as other static files of your project. In my case (NextJS), I placed it under public/sql-wasm.wasm. You then need to tell sql.js where to find it:

// in service-worker.js

// ...
// will automatically fetch /sql-wasm.wasm
const SQL = await initSqlJs({ locateFile: () => "/sql-wasm.wasm" });
// ...

4. absurd-sql will try to call postMessage from the global self object: it doesn't exist in ServiceWorker global scope, you need to provide it:

// in service-worker.js

async function postMessage(message) {
  const clients = await self.clients.matchAll({ type: "window" });
  for (const client of clients) {
    client.postMessage(message);
  }
}

// this must be added before calling absurd-sql functions
self.postMessage = postMessage;

5. Contrary to when using it in a basic WebWorker, the db initialization should only be done once in a ServiceWorker (on the activate lifecycle for example), you can then use it wherever you want:

let db;

self.addEventListener("activate", async function (event) {
  // ...

  db = await initDB();
});

async function initDB() {
  const SQL = await initSqlJs({ locateFile: () => "/sql-wasm.wasm" });
  const sqlFS = new SQLiteFS(SQL.FS, new IndexedDBBackend());
  SQL.register_for_idb(sqlFS);

  SQL.FS.mkdir("/sql");

  try {
    SQL.FS.mount(sqlFS, {}, "/sql");
  } catch (e) {
    console.error("mount already exists");
  }

  const path = "/sql/db.sqlite";
  if (typeof SharedArrayBuffer === "undefined") {
    const stream = SQL.FS.open(path, "a+");
    await stream.node.contents.readIfFallback();
    SQL.FS.close(stream);
  }

  const db = new SQL.Database(path, { filename: true });
  db.exec(`
    PRAGMA page_size=8192;
    PRAGMA journal_mode=MEMORY;
  `);

  return db;
}

Hope this will help some of you. Also, ask me if you want some more details about my NextJS - absurd-sql specific setup.

[billymoon]

@mdubourg001 can you share some more details about the next js specific parts of your setup?

[mdubourg001]

Yes, in Next you actually have to update your next.config.js in order to use the workbox-webpack-plugin:

npm install workbox-webpack-plugin -D

in next.config.js:

const { InjectManifest } = require("workbox-webpack-plugin");

module.exports = {
  // ...
  webpack: (config, options) => {
    // ...

    if (!options.isServer) {
      config.plugins.push(
        new InjectManifest({
          // update this with the actual location on your serviceWorker source file
          swSrc: "./src/service-worker/serviceWorker.ts",
          swDest: "../public/sw.js",
          include: ["__nothing__"],
        })
      );
    }

	// ...

    return config;
  },
};

And that's it ! 😅 Do you have any specific issue with this in Next @billymoon ?

[oceanwap]

@mdubourg001
I am getting below error when I run it in service worker