I was asked hours ago by a concurrency enthusiast, whose website has been a valuable source of information for me on the topic, to tell in a sentence what capability Asyncpal provides to users above the Python standard library (stdlib).

I found the question interesting because it goes straight to the point and calls for a concise answer. I believe the answer is missing from this 'Show HN'. Here is my response to the question:

Asyncpal unifies the stdlib (concurrent.futures + multiprocessing.pool) and provides true elastic pools (grow + shrink) with an intuitive interface.

If I'm given a chance to develop, I would start with the last part of the sentence that might sound subjective. For example, the 'Future' class in 'concurrent.futures' exposes a method named 'result' to collect the result of a task. In contrast, the 'Future' class in Asyncpal exposes a 'collect' method and a 'result' property.

The stdlib's pools only grow in size and do not shrink, making them non-elastic. Therefore, I would not want to keep them alive in the background for sporadic workloads [1].

Discussions on 'concurrent.futures' vs 'multiprocessing.pool.Pool' highlight that each has unique features. While 'concurrent.futures' is the modern package, it omits some niceties found in 'multiprocessing.pool.Pool'. For example, 'concurrent.futures' has only one 'map' [2] method, which works eagerly and therefore not suitable for very long iterables [3][4]. However, I acknowledge the superiority of 'concurrent.futures.Future' [5] over 'multiprocessing.pool.AsyncResult' [6] because tasks cannot be cancelled with the latter (among other things).

[1] https://www.cloudcomputingpatterns.org/unpredictable_workloa... (related)

[2] https://docs.python.org/3/library/concurrent.futures.html#co...

[3] https://docs.python.org/3/library/multiprocessing.html#multi...

[4] https://docs.python.org/3/library/multiprocessing.html#multi...

[5] https://docs.python.org/3/library/concurrent.futures.html#fu...

[6] https://docs.python.org/3/library/multiprocessing.html#multi...

Looks very well designed. As someone new to asynchronous programming in python, I have been using asyncio.to_thread in a task. Although the best way to manage things is to use a thread pool executor

Thanks for your comment !

When you are doing parallelism, don't forget to protect the 'entry point' of the program by using "if __name__ == '__main__'", and also avoid the __main__.py file [1]

  # this file isn't `__main__.py` !
  from asyncpal import ProcessPool

  def square(x):
      return x**2

  if __name__ == "__main__":  # very important !
      with ProcessPool(4) as pool:
          numbers = range(1000)
          # note that 'map_all' isn't lazy
          iterator = pool.map(square, numbers)  # map is lazy
          result = tuple(iterator)
          assert result == tuple(map(square, numbers))

[1] https://discuss.python.org/t/why-does-multiprocessing-not-wo...

that's cool. but i'm looking for a preemptive concurrency like golang in pyhton that schedules routines and get threads back when routine runs a blocking syscall. the hard part is detection of syscalls. is there any idea?

If I understand correctly, goroutines = asyncio (non-invasive) + asyncio.to_thread (when needed, but automatic). So, it is still cooperative concurrency that feels preemptive due to its design. This Go capability is interesting, and it 'seems' that it cannot be replicated without some integration with the runtime of the target language, i.e., a 'baked-in solution'. For now, I'm fine with true preemptive concurrency, but this might change in the future.

This is sweet. I will be using this going forward. Love the simplicity of managing everything

Thank you !