Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run absurd-sql in a serviceworker #54

Open
klehmann opened this issue Jun 1, 2022 · 6 comments
Open

Run absurd-sql in a serviceworker #54

klehmann opened this issue Jun 1, 2022 · 6 comments

Comments

@klehmann
Copy link

klehmann commented Jun 1, 2022

I did some quick tests to run absurd-sql in a serviceworker, but it fails like this:

serviceworker.js:63 Failed to open the database Yb.w.ErrnoError.w.ErrnoError {node: undefined, Oa: 20, message: 'FS error', hd: ƒ}Oa: 20hd: ƒ (c)message: "FS error"node: undefined[[Prototype]]: Error
at Object.Yb (webpack:https://absurd-example-project/./node_modules/@jlongster/sql.js/dist/sql-wasm.js?:160:231)
at Object.lc (webpack:https://absurd-example-project/./node_modules/@jlongster/sql.js/dist/sql-wasm.js?:160:402)
at eval (webpack:https://absurd-example-project/./node_modules/@jlongster/sql.js/dist/sql-wasm.js?:177:15)
at new Promise ()
at initSqlJs (webpack:https://absurd-example-project/./node_modules/@jlongster/sql.js/dist/sql-wasm.js?:24:24)
at openDB (webpack:https://absurd-example-project/./src/serviceworker.js?:19:77)
at eval (webpack:https://absurd-example-project/./src/serviceworker.js?:47:20)
eval @ serviceworker.js:63

Before diving deeper I would like to know if this is something that is supposed to work in general. My idea was to serve static HTML/JS files and images out of an absurd-sql instance and add some sync capabilities to fetch them from a server-side Sqlite DB.
But there's probably a workaround in using the Worker from the sample project and store duplicates of the web resources in the Cache Storage API.

@tantaman
Copy link

Is the idea that the service worker would intercept calls to your server and serve whatever it has locally in the db?

random observation on the particular use case -- I'm not sure sqlite is the best fit for storing images and static files.

@klehmann
Copy link
Author

Yes, that’s the idea, to have a web application that is synced onto the device/desktop browser and can run offline. When online, app data and HTML/JS is synced with the server.

@mdubourg001
Copy link

After trying hard for a few hours, I finally managed to run absurd-sql in a ServiceWorker (for a NextJS app but not important here), here's how I did it:

  1. initBackend wants a Worker instance in parameter. In the case of a ServiceWorker, the Worker object is navigator.serviceWorker.controller:
if (navigator.serviceWorker.controller) {
	const worker = navigator.serviceWorker.controller;
	initBackend(worker);
}
  1. you need to tell Webpack not to resolve requires of native node modules (fs, path, crypto) (related to issue can't resolve fs #27):
// webpack.config.js

module.exports = {
	// ...
	resolve: {
		fallback: {
			crypto: false,
          	path: false,
          	fs: false,
		}
	}
}
  1. absurd-sql needs to download a WASM binary in order to work (this is something that took me some time to figure out): this is not really documented but the binary can be found in the example repository src/examples/sql-wasm.wasm, and must be placed in order to be served as other static files of your project. In my case (NextJS), I placed it under public/sql-wasm.wasm. You then need to tell sql.js where to find it:
// in service-worker.js

// ...
// will automatically fetch /sql-wasm.wasm
const SQL = await initSqlJs({ locateFile: () => "/sql-wasm.wasm" });
// ...
  1. absurd-sql will try to call postMessage from the global self object: it doesn't exist in ServiceWorker global scope, you need to provide it:
// in service-worker.js

async function postMessage(message) {
  const clients = await self.clients.matchAll({ type: "window" });
  for (const client of clients) {
    client.postMessage(message);
  }
}

// this must be added before calling absurd-sql functions
self.postMessage = postMessage;
  1. Contrary to when using it in a basic WebWorker, the db initialization should only be done once in a ServiceWorker (on the activate lifecycle for example), you can then use it wherever you want:
let db;

self.addEventListener("activate", async function (event) {
  // ...

  db = await initDB();
});

async function initDB() {
  const SQL = await initSqlJs({ locateFile: () => "/sql-wasm.wasm" });
  const sqlFS = new SQLiteFS(SQL.FS, new IndexedDBBackend());
  SQL.register_for_idb(sqlFS);

  SQL.FS.mkdir("/sql");

  try {
    SQL.FS.mount(sqlFS, {}, "/sql");
  } catch (e) {
    console.error("mount already exists");
  }

  const path = "/sql/db.sqlite";
  if (typeof SharedArrayBuffer === "undefined") {
    const stream = SQL.FS.open(path, "a+");
    await stream.node.contents.readIfFallback();
    SQL.FS.close(stream);
  }

  const db = new SQL.Database(path, { filename: true });
  db.exec(`
    PRAGMA page_size=8192;
    PRAGMA journal_mode=MEMORY;
  `);

  return db;
}

Hope this will help some of you. Also, ask me if you want some more details about my NextJS - absurd-sql specific setup.

@billymoon
Copy link

@mdubourg001 can you share some more details about the next js specific parts of your setup?

@mdubourg001
Copy link

Yes, in Next you actually have to update your next.config.js in order to use the workbox-webpack-plugin:

npm install workbox-webpack-plugin -D

in next.config.js:

const { InjectManifest } = require("workbox-webpack-plugin");

module.exports = {
  // ...
  webpack: (config, options) => {
    // ...

    if (!options.isServer) {
      config.plugins.push(
        new InjectManifest({
          // update this with the actual location on your serviceWorker source file
          swSrc: "./src/service-worker/serviceWorker.ts",
          swDest: "../public/sw.js",
          include: ["__nothing__"],
        })
      );
    }

	// ...

    return config;
  },
};

And that's it ! 😅 Do you have any specific issue with this in Next @billymoon ?

@oceanwap
Copy link

oceanwap commented Mar 25, 2023

@mdubourg001
I am getting below error when I run it in service worker

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants