[FEATURE] Add examples in repo for long-running workers

[dec0dOS]

Duplicate issue

• I've searched for any related issues and avoided creating a duplicate issue.

Description

Hello! I just wanted to take a moment to express my sincere gratitude for your fantastic project – it's exactly what I've been searching for all this time! I've come across many worker thread wrappers, but none of them seem to fit my specific needs.

I'd like to suggest adding examples to the repository that showcase the usage of long-running workers. I believe this would greatly benefit users like me who are looking to understand how to implement this feature effectively.

I'm looking to execute long-running I/O and CPU intensive tasks that need to communicate with each other using event-based communication. These tasks are loaded from a configuration file at startup and may run for the entire duration of the application. The number of workers won't change during the app's execution, and each task has its own unique configuration.

Most worker thread wrappers are designed for fixed or dynamic pools, which don't align with my use case. However, it seems that Poolifier is architected differently and might be suitable for my needs. I'm just a bit unsure about how to use it correctly.

It seems more logical to use Poolifier compared to raw worker threads, especially considering this issue: #801. Once that's resolved, transitioning from worker threads to a cluster would be a breeze.

Since Poolifier supports MessageChannel, I'm curious if it's possible to utilize BroadcastChannel for worker threads within Poolifier for my specific use case.

Could you please provide some clarification on whether Poolifier is indeed suitable for my needs? Your guidance would be greatly appreciated!

Attachments

No response

[jerome-benoit]

I'm looking to execute long-running I/O and CPU intensive tasks that need to communicate with each other using event-based communication. These tasks are loaded from a configuration file at startup and may run for the entire duration of the application. The number of workers won't change during the app's execution, and each task has its own unique configuration.

It's an ongoing reflection: #596

But basically:

• Fixed pool
• No tasks queueing
• RR tasks distribution
• Number of tasks = number of workers for service

The service running use case differs a lot from tasks with request/response.
The examples show how to do it for HTTP servers. Clients use-case will work the same way but you can imagine running several clients in one worker.

Since Poolifier supports MessageChannel, I'm curious if it's possible to utilize BroadcastChannel for worker threads within Poolifier for my specific use case.

Poolifier internal communication != task(s) internal communication

The way you design your application communication between task(s) running instance in workers is out of the poolifier scope and you can do it the way you want.

See for example https://github.com/SAP/e-mobility-charging-stations-simulator

[dec0dOS]

Thank you for your response, @jerome-benoit!

If I understand you correctly, since the issue #596 hasn't been resolved yet, it seems that Poolifier may not be the best fit for my use case. In this case, I should consider using worker threads directly or creating my own wrapper. Please correct me if I'm mistaken.

Based on your guidance, it looks like the example you provided uses raw worker threads instead. https://github.com/SAP/e-mobility-charging-stations-simulator/blob/main/src/worker/WorkerSet.ts#L4

[dec0dOS]

Poolifier internal communication != task(s) internal communication
The way you design your application communication between task(s) running instance in workers is out of the poolifier scope and you can do it the way you want.

Initially, I thought that Poolifier handled the complexities of synchronizing states between workers, such as using MessageChannel for worker threads or IPC for cluster. I'm somewhat surprised to find that there doesn't appear to be a readily available solution for this specific use case within the Node.js ecosystem.

[jerome-benoit]

Thank you for your response, @jerome-benoit!

If I understand you correctly, since the issue #596 hasn't been resolved yet, it seems that Poolifier may not be the best fit for my use case. In this case, I should consider using worker threads directly or creating my own wrapper. Please correct me if I'm mistaken.

The issue quoted will hardly end offering out of box solution for all kind of tasks running as background processes. Each task can have its own internals that can make their moving, spawning across workers very specific.
For example:

• WebSockets server: probably the number of incoming open connection, no moving possible (stateful connection)
• WebSocket client pool: probably the number of outgoing open and used connection, moving possible if unused
• ...

Designing and implementing an API for that kind of tasks requires quite a lot of plumbing first, and a way to write an own worker selection fitting the task internals. And we are still unsure about incorporating it directly in poolifier, or as plugins to worker choice strategies.

We are taking PRs :)

Based on your guidance, it looks like the example you provided uses raw worker threads instead. https://github.com/SAP/e-mobility-charging-stations-simulator/blob/main/src/worker/WorkerSet.ts#L4

Both are working depending on the use case and its related workload.

[jerome-benoit]

Poolifier internal communication != task(s) internal communication
The way you design your application communication between task(s) running instance in workers is out of the poolifier scope and you can do it the way you want.

Initially, I thought that Poolifier handled the complexities of synchronizing states between workers, such as using MessageChannel for worker threads or IPC for cluster. I'm somewhat surprised to find that there doesn't appear to be a readily available solution for this specific use case within the Node.js ecosystem.

There's a bunch of npm packages to do IPC. But you are talking about read/write atomically on shared data structure from various processes. The Node.js solution for worker_threads is SharedArrayBuffer and Atomics JS API. There's no equivalent for cluster (or child_process).

[dec0dOS]

Hello, @jerome-benoit!

Thank you for your response. I've been diving deep into worker threads and the cluster API lately, and it seems like the missing piece of the puzzle might be the cluster API with polyfills that fill in the gaps from the worker threads API. This could provide a more seamless way to implement worker threads-like functionality within the cluster, making it easier to use the cluster in the same way as worker threads.

Initially, I thought Poolifier might do just that, especially since effective communication between tasks is crucial for a Pool implementation. Having a solution like this in place would simplify Poolifier significantly.

However, upon further exploration of worker thread implementation, I've noticed that it requires various workarounds and hacks to function as intended. You can check out these discussions on GitHub: Issue 13143 and Issue 41673. It appears that MessageChannels are also facing performance issues, which I'm not the only one to observe (Issue 767 and PR 5302).

So, for applications with long-running, fixed numbers of tasks spawned at startup, the cluster API (or just using directly the child_process.fork()) seems to be a more robust choice under the hood.

In my opinion, a solution like "cluster-plus-plus" should be developed as a separate package, rather than being directly within the scope of the Poolifier project. This way, anyone could replace the existing worker threads implementation with "cluster-plus-plus," and Poolifier could potentially use it as well.

For interprocess communication, an embedded database that doesn't rely on third-party dependencies like Redis could be a suitable replacement for SharedArrayBuffer. I've taken a liking to lmdb-js at first glance. Additionally, libraries like ZeroMQ or pm2-axon could fill the gap left by MessageChannel/MessagePort/BroadcastChannel.

I would greatly appreciate hearing your thoughts on this matter. I'm considering starting the development of a solution as described above, as it could be a valuable addition to the Node.js ecosystem. Your expertise, as a maintainer of one of the top Node.js pool libraries who deals with worker threads and clusters regularly, would be highly valuable. Your input would be greatly appreciated.

[jerome-benoit]

I would greatly appreciate hearing your thoughts on this matter. I'm considering starting the development of a solution as described above, as it could be a valuable addition to the Node.js ecosystem. Your expertise, as a maintainer of one of the top Node.js pool libraries who deals with worker threads and clusters regularly, would be highly valuable. Your input would be greatly appreciated.

It might take a bit of time before I will be able to answer to all points. But for more insight on the status of parallelism with JS, you have to keep in mind that it has been designed as a mono-threaded runtime interpreter (v8, JavaScriptCore, ...), being integrated in node.js or browsers or bun or deno or .... Given that initial design, any subsequent work done at introducing multi-processing or multi-threading in JS, even the well-thought one, remains hackish and not that performant.
If you compare for example with rust, go or even java, wandering on the parallelism path with JS is way less easy and the end-results are not always as performant as you might expect, because of the initial design choice.

[jerome-benoit]

So, for applications with long-running, fixed numbers of tasks spawned at startup, the cluster API (or just using directly the child_process.fork()) seems to be a more robust choice under the hood.

In my opinion, a solution like "cluster-plus-plus" should be developed as a separate package, rather than being directly within the scope of the Poolifier project. This way, anyone could replace the existing worker threads implementation with "cluster-plus-plus," and Poolifier could potentially use it as well.

For interprocess communication, an embedded database that doesn't rely on third-party dependencies like Redis could be a suitable replacement for SharedArrayBuffer. I've taken a liking to lmdb-js at first glance. Additionally, libraries like ZeroMQ or pm2-axon could fill the gap left by MessageChannel/MessagePort/BroadcastChannel.

• worker_threads is designed to mimic the POSIX threads features. The multithreading model apply here.
• child_process is designed to mimic POSIX fork(). The multiprocessing model apply here. Cluster module is using that model.

If you want the tackle the known issues with the current implementations allowing parallel computing in node.js:

• worker_threads IPC (message channel, broadcast channel) has scalability issue. If you want to address it, I suggest doing it upstream directly by proposing a scalable implementation.
• worker_threads high memory usage. It's probably a work to be done at the interpreter level.
• issues left hindering worker_threads usage and forcing the use of child_process instead.
• worker_threads shared memory encapsulation and handling into a more friendly API.

The multiprocessing model in general has too much drawbacks per design to tackle the parallel computing paradigm. The node.js ecosystem is more or less focusing at making worker_threads as mature as child_process.
For example, the cluster module with worker_threads would use less memory and CPU than the current one using child_process while offering shared memory.

[jerome-benoit]

It's a discussion, not an issue to track. If the discussion needs to continue, please do it at https://github.com/poolifier/poolifier/discussions

[dec0dOS]

Oh, I'm sorry. I thought you had received my reply. It seems that GitHub might have lost it. I'm currently experiencing some issues while using GitHub.