VS Code python unittest fails to import modules outside tests folder during debug

[cfframe]

Type: Bug

Since earlier this month, attempting to debug Python unittest tests in existing and new projects fails with an error similar to

ModuleNotFoundError: No module named '<file or folder name>'

for any test file trying to import something outside of the tests folder.

A basic project based on guidance in Testing Python in Visual Studio Code can be cloned from https://github.com/cfframe/PythonBasicTest.

Tests will run using the gui, but debugging a test generates this error in the DEBUG CONSOLE:

E
======================================================================
ERROR: test_unittest (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_unittest
Traceback (most recent call last):
  File "C:\Python\3.10.11\lib\unittest\loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "c:\GitLab\cfframe\BasicTest3\PythonBasicTest\tests\test_unittest.py", line 1, in <module>
    import inc_dec as inc_dec    # The code to test
ModuleNotFoundError: No module named 'inc_dec'


----------------------------------------------------------------------
Ran 1 test in 0.009s

FAILED (errors=1)

Versions:

• VSCode versions 1.80.1, 1.80.0, 1.79.2, 1.79.1
• Python 3.10.11, 3.10.6, 3.9.13
• ms-python.python 2023.12.0
• ms-python.vscode-pylance 2023.7.20

I have disabled all other extensions to try troubleshooting this.

I have no similar problems with PyCharm on the same projects.

Extension version: 2023.12.0
VS Code version: Code 1.80.1 (74f6148eb9ea00507ec113ec51c489d6ffb4b771, 2023-07-12T17:22:07.651Z)
OS version: Windows_NT x64 10.0.19045
Modes:

System Info

Item	Value
CPUs	11th Gen Intel(R) Core(TM) i5-1145G7 @ 2.60GHz (8 x 2611)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok video_decode: enabled video_encode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: enabled
Load (avg)	undefined
Memory (System)	15.69GB (6.25GB free)
Process Argv	C:\GitLab\cfframe\BasicTest3\PythonBasicTest --crash-reporter-id 679c4d58-dda0-4be6-a133-afbb5e749e9c
Screen Reader	no
VM	0%

A/B Experiments

vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30445986
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
py29gd2263:30789496
vsclangdf:30486550
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
3biah626:30602489
pyind779:30671433
f6dab269:30613381
vscrp:30673768
pythonsymbol12:30671437
showlangstatbar:30737416
vsctsb:30748421
pythonms35:30701012
a2ce3375:30757347
pythonfmttext:30731395
pythoncmvfstrcf:30756944
fixshowwlkth:30771522
hideindicator:30785051
pythongtdpath:30769146
bgfeh915:30780428
pythonnosm12tcf:30779713
pythonidxpt:30784022
pythonnoceb:30776495
h7j2d465:30786200
dsvsc013cf:30789518

[eleanorjboyd]

Thank you for your issue report. We are looking into this now! In the meantime, you are likely on the new testing rewrite and this is why you saw a change in behavior since we just moved to this experiment to a larger group of people receiving it. You can opt out of the rewrite as I get this fix in by setting this in your user settings: "python.experiments.optOutFrom": ["pythonTestAdapter"],. If this doesn't work let me know as this will mean it is a different issue. Thank you and I will send updates in this thread as I get a fix in.

[eleanorjboyd]

@cfframe quick question, can you provide your settings for your workspace? To get them go to the command palette and then select settings.json for the workspace. I specifically need the ones regarding unittest args. Thanks

[cfframe]

Thanks @eleanorjboyd.

You can opt out of the rewrite as I get this fix in by setting this in your user settings: "python.experiments.optOutFrom": ["pythonTestAdapter"],

Adding that setting did not work.

I've now tested the latest few versions of the Python extension and found that v2023.8.0 allows test debugging to work so have reverted to that for now. Changing the Pylance version seems to make no difference, currently sticking with v2023.7.20.

Also found that Python v2023.10.0 and v2023.10.1 trigger a "Unittest Discovery Error", mentioning in case it is related.

[cfframe]

@eleanorjboyd

can you provide your settings for your workspace? To get them go to the command palette and then select settings.json for the workspace.

With python.experiments.optOutFrom removed:

{
    "python.testing.unittestArgs": [
        "-v",
        "-s",
        "./tests",
        "-p",
        "test_*.py"
    ],
    "python.testing.pytestEnabled": false,
    "python.testing.unittestEnabled": true
}

[eleanorjboyd]

Did you try the setting in your user settings as opposed to your workspace settings? Because this setting can only be applied in user settings in local window or in remote settings in remote window. Thanks

[cfframe]

Did you try the setting in your user settings as opposed to your workspace settings? Because this setting can only be applied in user settings in local window or in remote settings in remote window. Thanks

@eleanorjboyd
My bad, apologies - I had added it to the workspace settings. I've now added python.experiments.optOutFrom to my user settings, updated the Python extension to v2023.12.0 and can verify test debugging works.

[eleanorjboyd]

No worries! Glad it worked now, that helps me know where to look for the bug. Thanks and Ill keep you updated

[LoicH]

Thank you! I had the same issue with the extension v2023.12.0, I added "python.experiments.optOutFrom": ["pythonTestAdapter"] in my user settings, reloaded the windows, and now I can see my tests!

[eleanorjboyd]

@LoicH glad that works to get you working again, I will have a PR out for this feature soon to address the bug. Thanks

[cfframe]

@eleanorjboyd When will the fix for this be released? I've just trialled commenting out the "python.experiments.optOutFrom": ["pythonTestAdapter"] setting in my User settings file, now I have more recent versions of VS Code (1.81.0) and the Python extension (v2023.14.0). Debugging a unittest test still fails to find files in a project sub folder.

My folder structure:
PROJECT ROOT
-- src
-- -- python files
-- tests
-- -- test files

[eleanorjboyd]

Hi! Apologies this fix is not out in stable yet- it is just out in insiders. It will be released to stable next month or earlier if we do a point release. Thanks

[andreamah]

are there some verification steps for this? @eleanorjboyd

[AdrianB-sovo]

For those using WSL on Windows, the "python.experiments.optOutFrom" will not be supported on your local user settings, you should use the remote user settings.
ctrl + shift + p → "Preferences: Open Remote Settings (JSON)"

[eleanorjboyd]

Hello, is any able to verify that the fix worked? No worries if not as I can also have a team member verify before we send this out to stable. Thanks

[eleanorjboyd]

I can still repro, here are the steps

• Clone repo https://github.com/cfframe/PythonBasicTest
• Open in VS Code
• Select configure tests and configure to use unittest
• Run tests and verify one fails.
• Reload VS Code
• Open a python file, then open the tests folder
• Run tests once and then try to debug the first tets test_increment
• It will now fail
• Try to run the test (not debug) and verify its successful
• Now try to debug and it works

So, debugging only fails the first time.

E
======================================================================
ERROR: test_unittest (unittest.loader._FailedTest.test_unittest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_unittest
Traceback (most recent call last):
  File "/Users/donjayamanne/.pyenv/versions/3.11.0/lib/python3.11/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/donjayamanne/temp/PythonBasicTest/tests/test_unittest.py", line 1, in <module>
    import inc_dec    # The code to test
    ^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'inc_dec'


----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)

Show Logs

try these steps to verify

[cfframe]

Additional prior step: In User Settings, ensure that python.experiments.optOutFrom for PythonTestAdapter is either commented out or not present.

The symptoms have subtly changed.

Debugging fails all the time (see Outputs below) BUT:

• the first attempt generates an error
• subsequent attempts fail to stop on the break point

Note that you do not need to run the tests to get this behaviour, but you do need to click on the test icon to get the run/debug symbols to appear in the margin by the code.

Visual Studio Code version: 1.83.1 (user setup)
Python extension version: v2023.18.0

Outputs

First attempt:

E
======================================================================
ERROR: test_unittest (unittest.loader._FailedTest.test_unittest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_unittest
Traceback (most recent call last):
  File "C:\Python\3.11.3\Lib\unittest\loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\GitHub\PythonBasicTest\tests\test_unittest.py", line 1, in <module>
    import inc_dec    # The code to test
    ^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'inc_dec'


----------------------------------------------------------------------
Ran 1 test in 0.003s

FAILED (errors=1)

Second and subsequent attempts:

Finished running tests!

[eleanorjboyd]

Hello! The fix for this should be out in the pre-release of the python extension. Could you try any version that starts with v2023.19? If you switch to pre-release version of the extension this should be a version that starts with v2023.19. Thanks

I tried this one myself and did not see the reported errors in either scenario you outlined. Thanks

[lstolcman]

confirmed, v2023.18 does not work as explained by OP, pre-release 2023.19 works as expected

[eleanorjboyd]

Great, thank you for checking! Closing and marking as verified.

[bullgom]

Thank you for your issue report. We are looking into this now! In the meantime, you are likely on the new testing rewrite and this is why you saw a change in behavior since we just moved to this experiment to a larger group of people receiving it. You can opt out of the rewrite as I get this fix in by setting this in your user settings: "python.experiments.optOutFrom": ["pythonTestAdapter"],. If this doesn't work let me know as this will mean it is a different issue. Thank you and I will send updates in this thread as I get a fix in.

For those of you coming across this error while in a docker container,
You have to set this in your docker container's settings: Settings -> Remote[...Container...] -> add the setting

This happened out of the blue for me. And I was pulling my hair out for few days.
Thank for your help everyone.