testdir.runpytest with a not expression doesn't skip tests

[davidemoro]

There is a test that reproduces the problem:

• master...davidemoro:assert_outcomes_skipped

Can anyone verify if this is a runpytest bug or a problem of mine?

Code example:

@pytest.mark.parametrize("cli_options", [
    '-k "not test_skipped"',
    '-k dummy',
])
def test_capture_skipped(testdir, cli_options):
    testdir.makepyfile(r"""
        import pytest

        def test_skipped():
            assert True
        """)
    result = testdir.runpytest(cli_options)
    result.assert_outcomes(skipped=1, passed=0)

where:

(Pdb) result.outlines
['============================= test session starts ==============================', 'platform linux -- Python 3.6.1, pytest-3.5.1.dev67+gf25a17c8, py-1.5.3, pluggy-0.6.0', 'rootdir: /tmp/pytest-of-davide/pytest-25/test_capture_skipped0, inifile:', 'plugins: hypothesis-3.56.5', 'collected 1 item', '', 'test_capture_skipped.py .                                                [100%]', '', '=========================== 1 passed in 0.02 seconds ===========================', '']

Pip list:

$ pip list
Package        Version                        
-------------- -------------------------------
attrs          17.4.0                         
certifi        2018.4.16                      
chardet        3.0.4                          
coverage       4.5.1                          
hypothesis     3.56.5                         
idna           2.6                            
mock           2.0.0                          
more-itertools 4.1.0                          
nose           1.3.7                          
pbr            4.0.2                          
pip            10.0.1                         
pluggy         0.6.0                          
py             1.5.3                          
pytest         3.5.1.dev66+gff6f1014.d20180424
requests       2.18.4                         
setuptools     39.0.1                         
six            1.11.0                         
urllib3        1.22                           
wheel          0.31.0

OS version:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 17.04
Release:        17.04
Codename:       zesty

[pytestbot]

GitMate.io thinks possibly related issues are #1549 (junit output doesn't add skipped tests into tests total), #502 (py.test --doctest-modules doesn't skip setup.py), #357 (Selecting tests with -k doesn't work as expected), #748 (pytest skip: short test summary info does not display actual test file), and #2975 (Pytest 3.3: tests being skipped without failing).

[nicoddemus]

Hi @davidemoro,

Try passing *args instead of a single string argument:

import pytest

pytest_plugins = 'pytester'

@pytest.mark.parametrize("cli_options", [
    ('-k', "not test_skipped"),
    ('-k', 'dummy'),
])
def test_capture_skipped(testdir, cli_options):
    testdir.makepyfile(r"""
        import pytest

        def test_skipped():
            assert True
        """)
    result = testdir.runpytest(*cli_options)
    result.assert_outcomes(skipped=1, passed=0)

But your test still fails to me because in both cases the test won't be skipped, it will be deselected:

... snip
-------------------------------------- Captured stdout call ---------------------------------------
======================================= test session starts =======================================
platform win32 -- Python 3.6.3, pytest-3.5.1.dev110+g132fb61e, py-1.5.2, pluggy-0.6.1.dev
rootdir: C:\Users\Bruno\AppData\Local\Temp\pytest-of-Bruno\pytest-1213\test_capture_skipped1, inifile:
plugins: xdist-1.22.2, forked-0.2, hypothesis-3.38.9, myplugin-0.0.0
collected 1 item / 1 deselected

================================== 1 deselected in 0.01 seconds ===================================

HTH! 👍

[davidemoro]

Thanks for your help @nicoddemus, it worked fine with *cli_options.

Just a question: why deselected and not skipped?

[nicoddemus]

Hey @davidemoro,

"deselected" means not run because of some command-line option "deselected" a collected test, while "skipped" means a test was skipped by code (a marker or pytest.skip).

[nicoddemus]

Closing now, feel free to follow up with any other questions. 👍

[davidemoro]

Thanks, I got it!