Multi-line setup.cfg configs break command expansion

[albertodonato]

Issue

Since switching to version 4 (4.2.6 specifically) from version 3, tox is treating multi-line entries in setup.cfg differently, as they seem to be expanded over mulitple lines when the var is used in a command (see example below).

Environment

• OS: Ubuntu 22.10 (kinetic)
• pip list of the host Python where tox is installed:

$ pip list
Package       Version
------------- -------
cachetools    5.2.1
chardet       5.1.0
colorama      0.4.6
distlib       0.3.6
filelock      3.9.0
packaging     23.0
pip           22.3.1
platformdirs  2.6.2
pluggy        1.0.0
pyproject_api 1.4.0
setuptools    59.6.0
tomli         2.0.1
tox           4.2.6
virtualenv    20.17.1

Output of running tox

Provide the output of tox -rvv:

$ tox -rvv
sample: 90 W remove tox env folder /home/ack/Desktop/tox-test/.tox/sample [tox/tox_env/api.py:321]
sample: 104 I find interpreter for spec PythonSpec(path=/home/ack/virtualenv/tox/bin/python3) [virtualenv/discovery/builtin.py:56]
sample: 104 D discover exe for PythonInfo(spec=CPython3.10.7.final.0-64, exe=/home/ack/virtualenv/tox/bin/python3, platform=linux, version='3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8) in /usr [virtualenv/discovery/py_info.py:437]
sample: 104 D filesystem is case-sensitive [virtualenv/info.py:24]
sample: 105 D got python info of /usr/bin/python3.10 from /home/ack/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.json [virtualenv/app_data/via_disk_folder.py:129]
sample: 105 I proposed PythonInfo(spec=CPython3.10.7.final.0-64, system=/usr/bin/python3.10, exe=/home/ack/virtualenv/tox/bin/python3, platform=linux, version='3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:63]
sample: 105 D accepted PythonInfo(spec=CPython3.10.7.final.0-64, system=/usr/bin/python3.10, exe=/home/ack/virtualenv/tox/bin/python3, platform=linux, version='3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:65]
sample: 118 I create virtual environment via CPython3Posix(dest=/home/ack/Desktop/tox-test/.tox/sample, clear=False, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:48]
sample: 118 D create folder /home/ack/Desktop/tox-test/.tox/sample/bin [virtualenv/util/path/_sync.py:9]
sample: 118 D create folder /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages [virtualenv/util/path/_sync.py:9]
sample: 118 D write /home/ack/Desktop/tox-test/.tox/sample/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
sample: 118 D   home = /usr/bin [virtualenv/create/pyenv_cfg.py:34]
sample: 118 D   implementation = CPython [virtualenv/create/pyenv_cfg.py:34]
sample: 118 D   version_info = 3.10.7.final.0 [virtualenv/create/pyenv_cfg.py:34]
sample: 118 D   virtualenv = 20.17.1 [virtualenv/create/pyenv_cfg.py:34]
sample: 118 D   include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
sample: 118 D   base-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
sample: 118 D   base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
sample: 118 D   base-executable = /usr/bin/python3.10 [virtualenv/create/pyenv_cfg.py:34]
sample: 119 D symlink /usr/bin/python3.10 to /home/ack/Desktop/tox-test/.tox/sample/bin/python [virtualenv/util/path/_sync.py:28]
sample: 119 D create virtualenv import hook file /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/_virtualenv.pth [virtualenv/create/via_global_ref/api.py:89]
sample: 119 D create /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/_virtualenv.py [virtualenv/create/via_global_ref/api.py:92]
sample: 119 D ============================== target debug ============================== [virtualenv/run/session.py:50]
sample: 119 D debug via /home/ack/Desktop/tox-test/.tox/sample/bin/python /home/ack/virtualenv/tox/lib/python3.10/site-packages/virtualenv/create/debug.py [virtualenv/create/creator.py:197]
sample: 119 D {
  "sys": {
    "executable": "/home/ack/Desktop/tox-test/.tox/sample/bin/python",
    "_base_executable": "/home/ack/Desktop/tox-test/.tox/sample/bin/python",
    "prefix": "/home/ack/Desktop/tox-test/.tox/sample",
    "base_prefix": "/usr",
    "real_prefix": null,
    "exec_prefix": "/home/ack/Desktop/tox-test/.tox/sample",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib/python310.zip",
      "/usr/lib/python3.10",
      "/usr/lib/python3.10/lib-dynload",
      "/home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0]",
  "makefile_filename": "/usr/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/usr/lib/python3.10/os.py'>",
  "site": "<module 'site' from '/usr/lib/python3.10/site.py'>",
  "datetime": "<module 'datetime' from '/usr/lib/python3.10/datetime.py'>",
  "math": "<module 'math' (built-in)>",
  "json": "<module 'json' from '/usr/lib/python3.10/json/__init__.py'>"
} [virtualenv/run/session.py:51]
sample: 133 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ack/.local/share/virtualenv) [virtualenv/run/session.py:55]
sample: 135 D install setuptools from wheel /home/ack/virtualenv/tox/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/setuptools-65.6.3-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
sample: 135 D install wheel from wheel /home/ack/virtualenv/tox/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/wheel-0.38.4-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
sample: 135 D install pip from wheel /home/ack/virtualenv/tox/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-22.3.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
sample: 136 D copy /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/distutils-precedence.pth to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/distutils-precedence.pth [virtualenv/util/path/_sync.py:36]
sample: 136 D copy directory /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.3.1-py3-none-any/pip to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/pip [virtualenv/util/path/_sync.py:36]
sample: 136 D copy directory /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/wheel [virtualenv/util/path/_sync.py:36]
sample: 136 D copy directory /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/_distutils_hack to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/_distutils_hack [virtualenv/util/path/_sync.py:36]
sample: 137 D copy directory /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/pkg_resources to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/pkg_resources [virtualenv/util/path/_sync.py:36]
sample: 139 D copy directory /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.dist-info to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/wheel-0.38.4.dist-info [virtualenv/util/path/_sync.py:36]
sample: 140 D copy /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.virtualenv to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/wheel-0.38.4.virtualenv [virtualenv/util/path/_sync.py:36]
sample: 141 D generated console scripts wheel wheel-3.10 wheel3 wheel3.10 [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
sample: 143 D copy directory /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/setuptools to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/setuptools [virtualenv/util/path/_sync.py:36]
sample: 158 D copy directory /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/setuptools-65.6.3.dist-info to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/setuptools-65.6.3.dist-info [virtualenv/util/path/_sync.py:36]
sample: 159 D copy /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-65.6.3-py3-none-any/setuptools-65.6.3.virtualenv to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/setuptools-65.6.3.virtualenv [virtualenv/util/path/_sync.py:36]
sample: 159 D generated console scripts  [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
sample: 169 D copy directory /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.3.1-py3-none-any/pip-22.3.1.dist-info to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/pip-22.3.1.dist-info [virtualenv/util/path/_sync.py:36]
sample: 169 D copy /home/ack/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-22.3.1-py3-none-any/pip-22.3.1.virtualenv to /home/ack/Desktop/tox-test/.tox/sample/lib/python3.10/site-packages/pip-22.3.1.virtualenv [virtualenv/util/path/_sync.py:36]
sample: 170 D generated console scripts pip3 pip-3.10 pip3.10 pip [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
sample: 170 I add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [virtualenv/run/session.py:61]
sample: 171 D write /home/ack/Desktop/tox-test/.tox/sample/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
sample: 171 D   home = /usr/bin [virtualenv/create/pyenv_cfg.py:34]
sample: 171 D   implementation = CPython [virtualenv/create/pyenv_cfg.py:34]
sample: 171 D   version_info = 3.10.7.final.0 [virtualenv/create/pyenv_cfg.py:34]
sample: 171 D   virtualenv = 20.17.1 [virtualenv/create/pyenv_cfg.py:34]
sample: 171 D   include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
sample: 171 D   base-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
sample: 171 D   base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
sample: 171 D   base-executable = /usr/bin/python3.10 [virtualenv/create/pyenv_cfg.py:34]
sample: 172 W commands[0]> .tox/sample/bin/python -c 'import sys; print(sys.argv[1:])' foo [tox/tox_env/api.py:427]
['foo']
sample: 182 I exit 0 (0.01 seconds) /home/ack/Desktop/tox-test> .tox/sample/bin/python -c 'import sys; print(sys.argv[1:])' foo pid=1627869 [tox/execute/api.py:275]
sample: 182 W commands[1]> bar [tox/tox_env/api.py:427]
sample: 183 C exit 2 (0.00 seconds) /home/ack/Desktop/tox-test> bar [tox/execute/api.py:275]
  sample: FAIL code 2 (0.09=setup[0.08]+cmd[0.01,0.00] seconds)
  evaluation failed :( (0.12 seconds)

For comparison, this is the output with the tox version in Ubuntu 22.10:

$ /usr/bin/tox -rvv 
using tox.ini: /home/ack/Desktop/tox-test/setup.cfg (pid 1628214)
  removing /home/ack/Desktop/tox-test/.tox/log
/usr/bin/python3 (/usr/bin/python3) is {'executable': '/usr/bin/python3', 'implementation': 'CPython', 'version_info': [3, 10, 7, 'final', 0], 'version': '3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0]', 'is_64': True, 'sysplatform': 'linux', 'extra_version_info': None}
sample uses /usr/bin/python3
using tox-3.21.4 from /usr/lib/python3/dist-packages/tox/__init__.py (pid 1628214)
skipping sdist step
sample start: getenv /home/ack/Desktop/tox-test/.tox/sample
sample cannot reuse: -r flag
sample create: /home/ack/Desktop/tox-test/.tox/sample
  removing /home/ack/Desktop/tox-test/.tox/sample
setting PATH=/home/ack/Desktop/tox-test/.tox/sample/bin:/home/ack/virtualenv/tox/bin:/home/ack/bin:/home/ack/system/bin:/home/ack/system/bash.d/bin:/home/ack/bin:/home/ack/system/bin:/home/ack/system/bash.d/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/home/ack/.local/bin:/home/ack/.cargo/bin:/home/ack/npm/bin:/home/ack/virtualenv/__main__/bin:/home/ack/go/bin:/home/ack/.local/bin:/home/ack/.cargo/bin:/home/ack/npm/bin:/home/ack/virtualenv/__main__/bin:/home/ack/go/bin
[1628217] /home/ack/Desktop/tox-test/.tox$ /usr/bin/python3 -m virtualenv --no-download --python /usr/bin/python3 sample
created virtual environment CPython3.10.7.final.0-64 in 60ms
  creator CPython3Posix(dest=/home/ack/Desktop/tox-test/.tox/sample, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ack/.local/share/virtualenv)
    added seed packages: pip==22.2, setuptools==59.6.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
sample finish: getenv /home/ack/Desktop/tox-test/.tox/sample after 0.16 seconds
sample start: finishvenv 
write config to /home/ack/Desktop/tox-test/.tox/sample/.tox-config1 as 'bfe5f8a30217c78e9b7c190244a7e314d8147b542f592ba783da0b27f87cf7fa /usr/bin/python3\n3.21.4 0 0 0'
sample finish: finishvenv  after 0.01 seconds
sample start: envreport 
setting PATH=/home/ack/Desktop/tox-test/.tox/sample/bin:/home/ack/virtualenv/tox/bin:/home/ack/bin:/home/ack/system/bin:/home/ack/system/bash.d/bin:/home/ack/bin:/home/ack/system/bin:/home/ack/system/bash.d/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/home/ack/.local/bin:/home/ack/.cargo/bin:/home/ack/npm/bin:/home/ack/virtualenv/__main__/bin:/home/ack/go/bin:/home/ack/.local/bin:/home/ack/.cargo/bin:/home/ack/npm/bin:/home/ack/virtualenv/__main__/bin:/home/ack/go/bin
[1628225] /home/ack/Desktop/tox-test$ /home/ack/Desktop/tox-test/.tox/sample/bin/python -m pip freeze >.tox/sample/log/sample-0.log
sample finish: envreport  after 0.47 seconds
sample installed: 
sample start: run-test-pre 
sample run-test-pre: PYTHONHASHSEED='1951728655'
sample finish: run-test-pre  after 0.00 seconds
sample start: run-test 
sample run-test: commands[0] | /home/ack/Desktop/tox-test/.tox/sample/bin/python -c 'import sys; print(sys.argv[1:])' foo bar baz
setting PATH=/home/ack/Desktop/tox-test/.tox/sample/bin:/home/ack/virtualenv/tox/bin:/home/ack/bin:/home/ack/system/bin:/home/ack/system/bash.d/bin:/home/ack/bin:/home/ack/system/bin:/home/ack/system/bash.d/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/home/ack/.local/bin:/home/ack/.cargo/bin:/home/ack/npm/bin:/home/ack/virtualenv/__main__/bin:/home/ack/go/bin:/home/ack/.local/bin:/home/ack/.cargo/bin:/home/ack/npm/bin:/home/ack/virtualenv/__main__/bin:/home/ack/go/bin
[1628227] /home/ack/Desktop/tox-test$ /home/ack/Desktop/tox-test/.tox/sample/bin/python -c 'import sys; print(sys.argv[1:])' foo bar baz
['foo', 'bar', 'baz']
sample finish: run-test  after 0.01 seconds
sample start: run-test-post 
sample finish: run-test-post  after 0.00 seconds
________________________________________________________________________________________________________________________________________________________________________________________________________________ summary _________________________________________________________________________________________________________________________________________________________________________________________________________________
  sample: commands succeeded
  congratulations :)

Minimal example

The following setup.cfg reproduces the issue:

[globals]
files_list =
    foo
    bar
    baz

[tox:tox]
envlist =
    sample
skipsdist = True

[testenv:sample]
commands =
    {envpython} -c 'import sys; print(sys.argv[1:])' {[globals]files_list}

Changing the definition as follows works with version 4, but is not backwards compatible:

[globals]
files_list =
    foo \
    bar \
    baz

[gaborbernat]

This is expected; we could document it as a breaking change, but that's about it.

[jugmac00]

This is expected; we could document it as a breaking change, but that's about it.

Why do we support line separation for e.g. passenv but not for this case?
cf #2676 (comment)

I just tried to use a comma as a separator... that won't work either.

❯ tox4
sample: commands[0]> .tox/sample/bin/python -c 'import sys; print(sys.argv[1:])' foo, bar, baz
['foo,', 'bar,', 'baz']
  sample: OK (0.16=setup[0.14]+cmd[0.02] seconds)
  congratulations :) (0.29 seconds)
  
  ❯ tox
sample create: /tmp/check2847/.tox/sample
sample run-test-pre: PYTHONHASHSEED='3897686535'
sample run-test: commands[0] | /tmp/check2847/.tox/sample/bin/python -c 'import sys; print(sys.argv[1:])' foo, bar, baz
['foo,', 'bar,', 'baz']
______________________________________________________________________________ summary ______________________________________________________________________________
  sample: commands succeeded
  congratulations :)

It looks like use space as a separator works for both tox 3 and tox 4.

❯ tox4
sample: commands[0]> .tox/sample/bin/python -c 'import sys; print(sys.argv[1:])' foo bar baz
['foo', 'bar', 'baz']
  sample: OK (0.06=setup[0.04]+cmd[0.02] seconds)
  congratulations :) (0.18 seconds)

❯ tox
sample run-test-pre: PYTHONHASHSEED='758776242'
sample run-test: commands[0] | /tmp/check2847/.tox/sample/bin/python -c 'import sys; print(sys.argv[1:])' foo bar baz
['foo', 'bar', 'baz']
_____________________________________________________________________________________________ summary _____________________________________________________________________________________________
  sample: commands succeeded
  congratulations :)

[albertodonato]

Yes, putting everything on one space-separated line is compatible with both cases.
It makes the config a bit ugly if you have a long list, though.

[gaborbernat]

Why do we support line separation for e.g. passenv but not for this case?
cf #2676 (comment)

Think about the type of configuration. pass_env is a list of keys, so you have one key per line. commands is a list of commands to run. One command per line, so no you cannot put one argument per line without using the trailing \ to mark them as lines that needs to be joined. This is the same logic as bash uses. These two are treated the same.