Prevent process pool inside of process pool when building tests

[sarahmarshy]

Description

When running 'mbed test' inside a virtual machine, I receive the following error:

Traceback (most recent call last):
  File "C:\ARM\thing\mbed-os\tools\test_api.py", line 2095, in build_test_worker
    bin_file = build_project(*args, **kwargs)
  File "C:\ARM\thing\mbed-os\tools\build_api.py", line 452, in build_project
    resources.inc_dirs)
  File "C:\ARM\thing\mbed-os\tools\toolchains\__init__.py", line 807, in compile_sources
    return self.compile_queue(queue, objects)
  File "C:\ARM\thing\mbed-os\tools\toolchains\__init__.py", line 831, in compile_queue
    p = Pool(processes=jobs_count)
  File "c:\python27\lib\multiprocessing\__init__.py", line 232, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild)
  File "c:\python27\lib\multiprocessing\pool.py", line 159, in __init__
    self._repopulate_pool()
  File "c:\python27\lib\multiprocessing\pool.py", line 223, in _repopulate_pool
    w.start()
  File "c:\python27\lib\multiprocessing\process.py", line 124, in start
    'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
Traceback (most recent call last):
  File "C:\ARM\thing\mbed-os\tools\test.py", line 217, in <module>
    build_profile=profile)
  File "C:\ARM\thing\mbed-os\tools\test_api.py", line 2191, in build_tests
    if not worker_result['result'] and not isinstance(worker_result['reason'], NotSupportedException):
KeyError: 'reason'
[ERROR] 'reason'
[mbed] ERROR: "python" returned error code 1.

The key error comes from unidentified exceptions from workers test_api.py#L2106, which doesn't create a 'reason' for failure entry in results if the exception is not known. I am guessing this is intentional so that the traceback will bubble up?

But the unidentified exception 'AssertionError' for 'daemonic processes are not allowed to have children' comes from the following:
test_api.py#L2142 , uses a variable, jobs_count (default is the number of cores on the machine), to create the number of worker processes in a pool, and these will handle the build system calls for all test projects. However, it is also used here in test_api.py#L2152, which tells the build system to use that many jobs when compiling source files. The build system will create a pool of processes to compile sources when the following check in toolchains/init.py#L806 is true. For tests, the number of sources to compile rarely exceeds 1 (as mbed-os is compiled once outside of the process pool and used as a library for all tests). It looks like the maximum it needs to compile is 3. Which is not a problem for 4 core machines, as the previously linked check will only create a pool when compiling if the number of sources to compile is greater than the number of jobs. However, if that is true, we create a process pool inside a process from the building test pool, which is by default disallowed in python. I ran into this issue on a virtual machine using 2 cores, which is probably why it never showed up.

This change prevents the build system from creating a pool when compiling sources for tests. There might be a hack in python to allow pools within pools, but this is what I have for now.

Status

READY

Steps to test or reproduce

With a clean build folder: Run mbed test -m x -t x --jobs 2

@bridadan

[bridadan]

Awesome, great find @sarahmarshy! @jeromecoutant I'm pretty sure this is the problem you've been experiencing! Could you pull it down and see if it fixes your issue?

[bridadan]

/morph test

[mbed-bot]

Result: SUCCESS

Your command has finished executing! Here's what you wrote!

/morph test

Output

mbed Build Number: 1068

All builds and test passed!

[jeromecoutant]

Thx @bridadan and @sarahmarshy
Patch is approved on my side