Prevent process pool inside of process pool when building tests #3258
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When running 'mbed test' inside a virtual machine, I receive the following error:
The key error comes from unidentified exceptions from workers test_api.py#L2106, which doesn't create a 'reason' for failure entry in results if the exception is not known. I am guessing this is intentional so that the traceback will bubble up?
But the unidentified exception 'AssertionError' for 'daemonic processes are not allowed to have children' comes from the following:
test_api.py#L2142 , uses a variable, jobs_count (default is the number of cores on the machine), to create the number of worker processes in a pool, and these will handle the build system calls for all test projects. However, it is also used here in test_api.py#L2152, which tells the build system to use that many jobs when compiling source files. The build system will create a pool of processes to compile sources when the following check in toolchains/init.py#L806 is true. For tests, the number of sources to compile rarely exceeds 1 (as mbed-os is compiled once outside of the process pool and used as a library for all tests). It looks like the maximum it needs to compile is 3. Which is not a problem for 4 core machines, as the previously linked check will only create a pool when compiling if the number of sources to compile is greater than the number of jobs. However, if that is true, we create a process pool inside a process from the building test pool, which is by default disallowed in python. I ran into this issue on a virtual machine using 2 cores, which is probably why it never showed up.
This change prevents the build system from creating a pool when compiling sources for tests. There might be a hack in python to allow pools within pools, but this is what I have for now.
Status
READY
Steps to test or reproduce
With a clean build folder: Run
mbed test -m x -t x --jobs 2
@bridadan