Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multiprocessing.Pool in spawn mode breaks logging.handlers.QueueHandler configuration #121723

Closed
noreentry opened this issue Jul 13, 2024 · 6 comments
Assignees
Labels
type-bug An unexpected behavior, bug, or error

Comments

@noreentry
Copy link

noreentry commented Jul 13, 2024

Bug report

Bug description:

multiprocessing.Pool with spawn method breaks logging.handlers.QueueHandler configuration

import logging
import logging.config
import multiprocessing
import time


def _init(q):
    logging.config.dictConfig({
        'version': 1,
        'disable_existing_loggers': True,
        'handlers': {'log_to_parent': {'class': 'logging.handlers.QueueHandler', 'queue': q}},
        'root': {'handlers': ['log_to_parent'], 'level': 'DEBUG'}
    })
    logging.getLogger().info('log from child')


if __name__ == '__main__':    
    multiprocessing.set_start_method('spawn')
    with multiprocessing.Manager() as manager:
        # q = manager.Queue()
        q = multiprocessing.Queue()
        
        # listen for log messages from child processes
        # ...

        with multiprocessing.Pool(processes=1, maxtasksperchild=1, initializer=_init, initargs=(q,)) as pool:
            time.sleep(1)

got exception (when q = manager.Queue() or q = multiprocessing.Queue())

Process SpawnPoolWorker-2:
Traceback (most recent call last):
  File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/logging/config.py", line 581, in configure
    handler = self.configure_handler(handlers[name])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/logging/config.py", line 792, in configure_handler
    proxy_queue = MM().Queue()
                  ^^^^
  File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/multiprocessing/managers.py", line 562, in start
    self._process.start()
  File "/home/foo/miniconda3/envs/py312forge/lib/python3.12/multiprocessing/process.py", line 118, in start
    assert not _current_process._config.get('daemon'), \
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: daemonic processes are not allowed to have children

The above exception was the direct cause of the following exception:
...

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

@noreentry noreentry added the type-bug An unexpected behavior, bug, or error label Jul 13, 2024
@graingert graingert changed the title multirpocessing.Pool in spawn mode breaks logging.handlers.QueueHandler configuration multiprocessing.Pool in spawn mode breaks logging.handlers.QueueHandler configuration Jul 16, 2024
@picnixz
Copy link
Contributor

picnixz commented Jul 22, 2024

@vsajip I suspect this is caused by the fact that you instantiate a queue to get its runtime type. Maybe a protocol approach for a queue would be better? I don't know if the queue you expect should be any queue or if whatever is a queue-like API is fine.

@vsajip
Copy link
Member

vsajip commented Jul 22, 2024

A queue-like API is all that's needed, but by protocol approach you mean just checking for the existence of queue methods used by logging? The fundamental problem here is that a proxy is used from which you can't easily infer what type the proxy is to. Another, perhaps more generally useful way might be to expose the underlying runtime type on the proxy so that it can be readily queried.

@picnixz
Copy link
Contributor

picnixz commented Jul 22, 2024

A queue-like API is all that's needed, but by protocol approach you mean just checking for the existence of queue methods used by logging

Yes. Don't really care about the type itself, just checking that methods of certain names exist, not even their signatures. And if something goes wrong, there would be a TypeError when you want to use the queue I think. You could reduce issues by having a well-documented section on what kind of queue API you expect, and that should be enough I think.

@picnixz
Copy link
Contributor

picnixz commented Jul 22, 2024

Another, perhaps more generally useful way might be to expose the underlying runtime type on the proxy so that it can be readily queried

IIRC, the proxy is a local class and it does not necessarily require to have a class. It's a bit hard to explain here but if you look at multiprocessing.managers.BaseManager.register and how queue classes are registered in managers, you'll see that keeping track of the base type might be more tricky.

@vsajip
Copy link
Member

vsajip commented Jul 22, 2024

Right, I had a quick look when this issue came up before and it didn't seem that straightforward, which is why I went with the approach of instantiating the queue to get its type 😞

@picnixz
Copy link
Contributor

picnixz commented Jul 22, 2024

So... protocol approach it will be? (I can make a PR tomorrow or in two days)

@picnixz picnixz self-assigned this Jul 23, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Aug 2, 2024
…lers.QueueHandler`. (pythonGH-122154)

(cherry picked from commit fb864c7)

Co-authored-by: Bénédikt Tran <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Aug 2, 2024
…lers.QueueHandler`. (pythonGH-122154)

(cherry picked from commit fb864c7)

Co-authored-by: Bénédikt Tran <[email protected]>
vsajip pushed a commit that referenced this issue Aug 2, 2024
vsajip pushed a commit that referenced this issue Aug 2, 2024
@vsajip vsajip closed this as completed Aug 2, 2024
brandtbucher pushed a commit to brandtbucher/cpython that referenced this issue Aug 7, 2024
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Aug 13, 2024
…xt in emulated JIT CI (pythonGH-122969)

(cherry picked from commit 7b8328b)

Co-authored-by: Bénédikt Tran <[email protected]>
blhsing pushed a commit to blhsing/cpython that referenced this issue Aug 22, 2024
blhsing pushed a commit to blhsing/cpython that referenced this issue Aug 22, 2024
brandtbucher pushed a commit that referenced this issue Aug 22, 2024
…ext in emulated JIT CI (GH-122991)

(cherry picked from commit 7b8328b)
Co-authored-by: Bénédikt Tran <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type-bug An unexpected behavior, bug, or error
Projects
Development

No branches or pull requests

3 participants