Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set_env substiution regression in tox 4.x #2972

Open
living180 opened this issue Apr 4, 2023 · 8 comments
Open

set_env substiution regression in tox 4.x #2972

living180 opened this issue Apr 4, 2023 · 8 comments
Labels
help:wanted Issues that have been acknowledged, a solution determined and a PR might likely be accepted.

Comments

@living180
Copy link
Contributor

Issue

In tox 4.x, with a set_env key that contains a substitution from another section as well as an environment variable substitution, the values that should have been substituted from another section are lost. For more information, see the example below.

Environment

Provide at least:

OS: Linux

`pip list` of the host Python where `tox` is installed:
Package                       Version
----------------------------- --------------------
aiohttp                       3.8.4
aiosignal                     1.3.1
alabaster                     0.7.13
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
async-timeout                 4.0.2
attrs                         22.2.0
Babel                         2.12.1
beautifulsoup4                4.12.0
build                         0.10.0
cachetools                    5.3.0
certifi                       2022.12.7
cffi                          1.15.1
chardet                       5.1.0
charset-normalizer            3.1.0
check-manifest                0.49
click                         8.1.3
click-default-group           1.2.2
colorama                      0.4.6
covdefaults                   2.3.0
coverage                      7.2.2
defusedxml                    0.7.1
devpi-client                  6.0.3
devpi-common                  3.7.0
devpi-process                 0.3.0
devpi-server                  6.8.0
diff-cover                    7.5.0
distlib                       0.3.6
docutils                      0.19
editables                     0.3
exceptiongroup                1.1.1
execnet                       1.9.0
filelock                      3.10.7
flaky                         3.7.0
frozenlist                    1.3.3
furo                          2023.3.27
gitdb                         4.0.10
GitPython                     3.1.31
hatch-vcs                     0.3.0
hatchling                     1.14.0
hupper                        1.12
idna                          3.4
imagesize                     1.4.1
incremental                   22.10.0
iniconfig                     2.0.0
itsdangerous                  2.1.2
Jinja2                        3.1.2
lazy                          1.5
MarkupSafe                    2.1.2
multidict                     6.0.4
packaging                     23.0
passlib                       1.7.4
PasteDeploy                   3.0.1
pathspec                      0.11.1
pep517                        0.13.0
pip                           23.0.1
pkginfo                       1.9.6
plaster                       1.1.2
plaster-pastedeploy           1.0.1
platformdirs                  3.2.0
pluggy                        1.0.0
psutil                        5.9.4
py                            1.11.0
pycparser                     2.21
Pygments                      2.14.0
pyproject_api                 1.5.1
pyproject_hooks               1.0.0
pyramid                       2.0.1
pytest                        7.2.2
pytest-cov                    4.0.0
pytest-mock                   3.10.0
pytest-xdist                  3.2.1
python-dateutil               2.8.2
re-assert                     1.1.0
regex                         2023.3.23
repoze.lru                    0.7
requests                      2.28.2
ruamel.yaml                   0.17.21
ruamel.yaml.clib              0.2.7
setuptools                    67.5.1
setuptools-scm                7.1.0
six                           1.16.0
smmap                         5.0.0
snowballstemmer               2.2.0
soupsieve                     2.4
Sphinx                        6.1.3
sphinx_argparse_cli           1.11.0
sphinx_autodoc_typehints      1.22
sphinx-basic-ng               1.0.0b1
sphinx-copybutton             0.5.1
sphinx_inline_tabs            2022.1.2b11
sphinxcontrib-applehelp       1.0.4
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
sphinxcontrib-towncrier       0.3.2a0
strictyaml                    1.7.3
time-machine                  2.9.0
tomli                         2.0.1
towncrier                     22.12.0
tox                           4.4.9.dev3+g8a7327c1
translationstring             1.4
trove-classifiers             2023.3.9
typing_extensions             4.5.0
urllib3                       1.26.15
venusian                      3.0.0
virtualenv                    20.21.0
waitress                      2.1.2
WebOb                         1.8.7
wheel                         0.40.0
yarl                          1.8.2
zope.deprecation              5.0
zope.interface                6.0

Output of running tox

Provide the output of `tox -rvv`:
testenv: 269 W remove tox env folder /home/dharding/code/tox_test/.tox/testenv [tox/tox_env/api.py:322]
.pkg: 283 W remove tox env folder /home/dharding/code/tox_test/.tox/.pkg [tox/tox_env/api.py:322]
testenv: 305 I find interpreter for spec PythonSpec(path=/home/dharding/code/opensource/tox/.tox/dev/bin/python) [virtualenv/discovery/builtin.py:56]
testenv: 305 D discover exe for PythonInfo(spec=CPython3.10.10.final.0-64, exe=/home/dharding/code/opensource/tox/.tox/dev/bin/python, platform=linux, version='3.10.10 (main, Mar 25 2023, 18:12:59) [GCC 12.2.1 20230121]', encoding_fs_io=utf-8-utf-8) in /usr [virtualenv/discovery/py_info.py:437]
testenv: 305 D filesystem is case-sensitive [virtualenv/info.py:24]
testenv: 305 D got python info of /usr/bin/python3.10 from /home/dharding/.local/share/virtualenv/py_info/1/8a94588eda9d64d9e9a351ab8144e55b1fabf5113b54e67dd26a8c27df0381b3.json [virtualenv/app_data/via_disk_folder.py:129]
testenv: 333 I proposed PythonInfo(spec=CPython3.10.10.final.0-64, system=/usr/bin/python3.10, exe=/home/dharding/code/opensource/tox/.tox/dev/bin/python, platform=linux, version='3.10.10 (main, Mar 25 2023, 18:12:59) [GCC 12.2.1 20230121]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:63]
testenv: 334 D accepted PythonInfo(spec=CPython3.10.10.final.0-64, system=/usr/bin/python3.10, exe=/home/dharding/code/opensource/tox/.tox/dev/bin/python, platform=linux, version='3.10.10 (main, Mar 25 2023, 18:12:59) [GCC 12.2.1 20230121]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:65]
testenv: 351 I create virtual environment via CPython3Posix(dest=/home/dharding/code/tox_test/.tox/testenv, clear=False, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:48]
testenv: 352 D create folder /home/dharding/code/tox_test/.tox/testenv/bin [virtualenv/util/path/_sync.py:9]
testenv: 352 D create folder /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages [virtualenv/util/path/_sync.py:9]
testenv: 352 D write /home/dharding/code/tox_test/.tox/testenv/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
testenv: 352 D 	home = /usr/bin [virtualenv/create/pyenv_cfg.py:34]
testenv: 352 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:34]
testenv: 352 D 	version_info = 3.10.10.final.0 [virtualenv/create/pyenv_cfg.py:34]
testenv: 352 D 	virtualenv = 20.21.0 [virtualenv/create/pyenv_cfg.py:34]
testenv: 352 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
testenv: 352 D 	base-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
testenv: 352 D 	base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
testenv: 352 D 	base-executable = /usr/bin/python3.10 [virtualenv/create/pyenv_cfg.py:34]
testenv: 352 D symlink /usr/bin/python3.10 to /home/dharding/code/tox_test/.tox/testenv/bin/python [virtualenv/util/path/_sync.py:28]
testenv: 352 D create virtualenv import hook file /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/_virtualenv.pth [virtualenv/create/via_global_ref/api.py:89]
testenv: 352 D create /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/_virtualenv.py [virtualenv/create/via_global_ref/api.py:92]
testenv: 353 D ============================== target debug ============================== [virtualenv/run/session.py:50]
testenv: 353 D debug via /home/dharding/code/tox_test/.tox/testenv/bin/python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/virtualenv/create/debug.py [virtualenv/create/creator.py:193]
testenv: 353 D {
  "sys": {
    "executable": "/home/dharding/code/tox_test/.tox/testenv/bin/python",
    "_base_executable": "/home/dharding/code/tox_test/.tox/testenv/bin/python",
    "prefix": "/home/dharding/code/tox_test/.tox/testenv",
    "base_prefix": "/usr",
    "real_prefix": null,
    "exec_prefix": "/home/dharding/code/tox_test/.tox/testenv",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib/python310.zip",
      "/usr/lib/python3.10",
      "/usr/lib/python3.10/lib-dynload",
      "/home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.10.10 (main, Mar 25 2023, 18:12:59) [GCC 12.2.1 20230121]",
  "makefile_filename": "/usr/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/usr/lib/python3.10/os.py'>",
  "site": "<module 'site' from '/usr/lib/python3.10/site.py'>",
  "datetime": "<module 'datetime' from '/usr/lib/python3.10/datetime.py'>",
  "math": "<module 'math' from '/usr/lib/python3.10/lib-dynload/math.cpython-310-x86_64-linux-gnu.so'>",
  "json": "<module 'json' from '/usr/lib/python3.10/json/__init__.py'>"
} [virtualenv/run/session.py:51]
testenv: 370 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/dharding/.local/share/virtualenv) [virtualenv/run/session.py:55]
testenv: 372 D got embed update of distribution setuptools from /home/dharding/.local/share/virtualenv/wheel/3.10/embed/3/setuptools.json [virtualenv/app_data/via_disk_folder.py:129]
testenv: 373 D using periodically updated wheel /home/dharding/.local/share/virtualenv/wheel/house/setuptools-67.5.1-py3-none-any.whl [virtualenv/seed/wheels/periodic_update.py:41]
testenv: 374 D got embed update of distribution wheel from /home/dharding/.local/share/virtualenv/wheel/3.10/embed/3/wheel.json [virtualenv/app_data/via_disk_folder.py:129]
testenv: 374 D got embed update of distribution pip from /home/dharding/.local/share/virtualenv/wheel/3.10/embed/3/pip.json [virtualenv/app_data/via_disk_folder.py:129]
testenv: 375 D install setuptools from wheel /home/dharding/.local/share/virtualenv/wheel/house/setuptools-67.5.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
testenv: 375 D install wheel from wheel /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/wheel-0.38.4-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
testenv: 375 D install pip from wheel /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-23.0.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
testenv: 376 D copy /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.virtualenv to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/wheel-0.38.4.virtualenv [virtualenv/util/path/_sync.py:36]
testenv: 376 D copy /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/distutils-precedence.pth to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/distutils-precedence.pth [virtualenv/util/path/_sync.py:36]
testenv: 376 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.dist-info to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/wheel-0.38.4.dist-info [virtualenv/util/path/_sync.py:36]
testenv: 376 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/_distutils_hack to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/_distutils_hack [virtualenv/util/path/_sync.py:36]
testenv: 377 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/pip [virtualenv/util/path/_sync.py:36]
testenv: 377 D copy /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/setuptools-67.5.1.virtualenv to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/setuptools-67.5.1.virtualenv [virtualenv/util/path/_sync.py:36]
testenv: 378 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/setuptools to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/setuptools [virtualenv/util/path/_sync.py:36]
testenv: 379 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/wheel [virtualenv/util/path/_sync.py:36]
testenv: 386 D generated console scripts wheel-3.10 wheel3.10 wheel3 wheel [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
testenv: 409 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/pkg_resources to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/pkg_resources [virtualenv/util/path/_sync.py:36]
testenv: 415 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/setuptools-67.5.1.dist-info to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/setuptools-67.5.1.dist-info [virtualenv/util/path/_sync.py:36]
testenv: 417 D generated console scripts  [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
testenv: 432 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip-23.0.1.dist-info to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/pip-23.0.1.dist-info [virtualenv/util/path/_sync.py:36]
testenv: 432 D copy /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip-23.0.1.virtualenv to /home/dharding/code/tox_test/.tox/testenv/lib/python3.10/site-packages/pip-23.0.1.virtualenv [virtualenv/util/path/_sync.py:36]
testenv: 433 D generated console scripts pip3.10 pip3 pip-3.10 pip [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
testenv: 433 I add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [virtualenv/run/session.py:61]
testenv: 434 D write /home/dharding/code/tox_test/.tox/testenv/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
testenv: 434 D 	home = /usr/bin [virtualenv/create/pyenv_cfg.py:34]
testenv: 434 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:34]
testenv: 434 D 	version_info = 3.10.10.final.0 [virtualenv/create/pyenv_cfg.py:34]
testenv: 434 D 	virtualenv = 20.21.0 [virtualenv/create/pyenv_cfg.py:34]
testenv: 434 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
testenv: 434 D 	base-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
testenv: 434 D 	base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
testenv: 434 D 	base-executable = /usr/bin/python3.10 [virtualenv/create/pyenv_cfg.py:34]
.pkg: 445 I find interpreter for spec PythonSpec(path=/home/dharding/code/opensource/tox/.tox/dev/bin/python) [virtualenv/discovery/builtin.py:56]
.pkg: 445 D discover exe from cache /usr - exact False: PythonInfo({'architecture': 64, 'base_exec_prefix': '/usr', 'base_prefix': '/usr', 'distutils_install': {'data': '', 'headers': 'include/python3.10/UNKNOWN', 'platlib': 'lib/python3.10/site-packages', 'purelib': 'lib/python3.10/site-packages', 'scripts': 'bin'}, 'exec_prefix': '/usr', 'executable': '/home/dharding/code/opensource/tox/.tox/dev/bin/python', 'file_system_encoding': 'utf-8', 'has_venv': True, 'implementation': 'CPython', 'max_size': 9223372036854775807, 'original_executable': '/usr/bin/python3.10', 'os': 'posix', 'path': ['/home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/virtualenv/discovery', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/lib/python3.10/site-packages'], 'platform': 'linux', 'prefix': '/usr', 'real_prefix': None, 'stdout_encoding': 'utf-8', 'sysconfig': {'makefile_filename': '/usr/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile'}, 'sysconfig_paths': {'data': '{base}', 'include': '{installed_base}/include/python{py_version_short}{abiflags}', 'platlib': '{platbase}/{platlibdir}/python{py_version_short}/site-packages', 'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}', 'purelib': '{base}/lib/python{py_version_short}/site-packages', 'scripts': '{base}/bin', 'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}'}, 'sysconfig_scheme': None, 'sysconfig_vars': {'PYTHONFRAMEWORK': '', 'abiflags': '', 'base': '/usr', 'installed_base': '/usr', 'platbase': '/usr', 'platlibdir': 'lib', 'py_version_short': '3.10'}, 'system_executable': '/usr/bin/python3.10', 'system_stdlib': '/usr/lib/python3.10', 'system_stdlib_platform': '/usr/lib/python3.10', 'version': '3.10.10 (main, Mar 25 2023, 18:12:59) [GCC 12.2.1 20230121]', 'version_info': VersionInfo(major=3, minor=10, micro=10, releaselevel='final', serial=0), 'version_nodot': '310'}) [virtualenv/discovery/py_info.py:435]
.pkg: 446 I proposed PythonInfo(spec=CPython3.10.10.final.0-64, system=/usr/bin/python3.10, exe=/home/dharding/code/opensource/tox/.tox/dev/bin/python, platform=linux, version='3.10.10 (main, Mar 25 2023, 18:12:59) [GCC 12.2.1 20230121]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:63]
.pkg: 446 D accepted PythonInfo(spec=CPython3.10.10.final.0-64, system=/usr/bin/python3.10, exe=/home/dharding/code/opensource/tox/.tox/dev/bin/python, platform=linux, version='3.10.10 (main, Mar 25 2023, 18:12:59) [GCC 12.2.1 20230121]', encoding_fs_io=utf-8-utf-8) [virtualenv/discovery/builtin.py:65]
.pkg: 446 I create virtual environment via CPython3Posix(dest=/home/dharding/code/tox_test/.tox/.pkg, clear=False, no_vcs_ignore=False, global=False) [virtualenv/run/session.py:48]
.pkg: 447 D create folder /home/dharding/code/tox_test/.tox/.pkg/bin [virtualenv/util/path/_sync.py:9]
.pkg: 447 D create folder /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages [virtualenv/util/path/_sync.py:9]
.pkg: 447 D write /home/dharding/code/tox_test/.tox/.pkg/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
.pkg: 447 D 	home = /usr/bin [virtualenv/create/pyenv_cfg.py:34]
.pkg: 447 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:34]
.pkg: 447 D 	version_info = 3.10.10.final.0 [virtualenv/create/pyenv_cfg.py:34]
.pkg: 447 D 	virtualenv = 20.21.0 [virtualenv/create/pyenv_cfg.py:34]
.pkg: 447 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
.pkg: 447 D 	base-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
.pkg: 447 D 	base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
.pkg: 447 D 	base-executable = /usr/bin/python3.10 [virtualenv/create/pyenv_cfg.py:34]
.pkg: 447 D symlink /usr/bin/python3.10 to /home/dharding/code/tox_test/.tox/.pkg/bin/python [virtualenv/util/path/_sync.py:28]
.pkg: 447 D create virtualenv import hook file /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/_virtualenv.pth [virtualenv/create/via_global_ref/api.py:89]
.pkg: 447 D create /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/_virtualenv.py [virtualenv/create/via_global_ref/api.py:92]
.pkg: 447 D ============================== target debug ============================== [virtualenv/run/session.py:50]
.pkg: 448 D debug via /home/dharding/code/tox_test/.tox/.pkg/bin/python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/virtualenv/create/debug.py [virtualenv/create/creator.py:193]
.pkg: 447 D {
  "sys": {
    "executable": "/home/dharding/code/tox_test/.tox/.pkg/bin/python",
    "_base_executable": "/home/dharding/code/tox_test/.tox/.pkg/bin/python",
    "prefix": "/home/dharding/code/tox_test/.tox/.pkg",
    "base_prefix": "/usr",
    "real_prefix": null,
    "exec_prefix": "/home/dharding/code/tox_test/.tox/.pkg",
    "base_exec_prefix": "/usr",
    "path": [
      "/usr/lib/python310.zip",
      "/usr/lib/python3.10",
      "/usr/lib/python3.10/lib-dynload",
      "/home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages"
    ],
    "meta_path": [
      "<class '_virtualenv._Finder'>",
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "utf-8"
  },
  "version": "3.10.10 (main, Mar 25 2023, 18:12:59) [GCC 12.2.1 20230121]",
  "makefile_filename": "/usr/lib/python3.10/config-3.10-x86_64-linux-gnu/Makefile",
  "os": "<module 'os' from '/usr/lib/python3.10/os.py'>",
  "site": "<module 'site' from '/usr/lib/python3.10/site.py'>",
  "datetime": "<module 'datetime' from '/usr/lib/python3.10/datetime.py'>",
  "math": "<module 'math' from '/usr/lib/python3.10/lib-dynload/math.cpython-310-x86_64-linux-gnu.so'>",
  "json": "<module 'json' from '/usr/lib/python3.10/json/__init__.py'>"
} [virtualenv/run/session.py:51]
.pkg: 465 I add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/dharding/.local/share/virtualenv) [virtualenv/run/session.py:55]
.pkg: 466 D got embed update of distribution setuptools from /home/dharding/.local/share/virtualenv/wheel/3.10/embed/3/setuptools.json [virtualenv/app_data/via_disk_folder.py:129]
.pkg: 467 D using periodically updated wheel /home/dharding/.local/share/virtualenv/wheel/house/setuptools-67.5.1-py3-none-any.whl [virtualenv/seed/wheels/periodic_update.py:41]
.pkg: 468 D got embed update of distribution pip from /home/dharding/.local/share/virtualenv/wheel/3.10/embed/3/pip.json [virtualenv/app_data/via_disk_folder.py:129]
.pkg: 468 D got embed update of distribution wheel from /home/dharding/.local/share/virtualenv/wheel/3.10/embed/3/wheel.json [virtualenv/app_data/via_disk_folder.py:129]
.pkg: 468 D install setuptools from wheel /home/dharding/.local/share/virtualenv/wheel/house/setuptools-67.5.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
.pkg: 468 D install pip from wheel /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/pip-23.0.1-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
.pkg: 469 D install wheel from wheel /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/virtualenv/seed/wheels/embed/wheel-0.38.4-py3-none-any.whl via CopyPipInstall [virtualenv/seed/embed/via_app_data/via_app_data.py:47]
.pkg: 469 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/pip [virtualenv/util/path/_sync.py:36]
.pkg: 470 D copy /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/distutils-precedence.pth to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/distutils-precedence.pth [virtualenv/util/path/_sync.py:36]
.pkg: 470 D copy /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.virtualenv to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/wheel-0.38.4.virtualenv [virtualenv/util/path/_sync.py:36]
.pkg: 470 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/_distutils_hack to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/_distutils_hack [virtualenv/util/path/_sync.py:36]
.pkg: 471 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel-0.38.4.dist-info to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/wheel-0.38.4.dist-info [virtualenv/util/path/_sync.py:36]
.pkg: 471 D copy /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/setuptools-67.5.1.virtualenv to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/setuptools-67.5.1.virtualenv [virtualenv/util/path/_sync.py:36]
.pkg: 472 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/setuptools to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/setuptools [virtualenv/util/path/_sync.py:36]
.pkg: 473 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/wheel-0.38.4-py3-none-any/wheel to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/wheel [virtualenv/util/path/_sync.py:36]
.pkg: 480 D generated console scripts wheel-3.10 wheel3.10 wheel3 wheel [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
.pkg: 503 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/pkg_resources to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/pkg_resources [virtualenv/util/path/_sync.py:36]
.pkg: 510 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/setuptools-67.5.1-py3-none-any/setuptools-67.5.1.dist-info to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/setuptools-67.5.1.dist-info [virtualenv/util/path/_sync.py:36]
.pkg: 511 D generated console scripts  [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
.pkg: 525 D copy directory /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip-23.0.1.dist-info to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/pip-23.0.1.dist-info [virtualenv/util/path/_sync.py:36]
.pkg: 526 D copy /home/dharding/.local/share/virtualenv/wheel/3.10/image/1/CopyPipInstall/pip-23.0.1-py3-none-any/pip-23.0.1.virtualenv to /home/dharding/code/tox_test/.tox/.pkg/lib/python3.10/site-packages/pip-23.0.1.virtualenv [virtualenv/util/path/_sync.py:36]
.pkg: 526 D generated console scripts pip3 pip-3.10 pip pip3.10 [virtualenv/seed/embed/via_app_data/pip_install/base.py:41]
.pkg: 526 I add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [virtualenv/run/session.py:61]
.pkg: 527 D write /home/dharding/code/tox_test/.tox/.pkg/pyvenv.cfg [virtualenv/create/pyenv_cfg.py:30]
.pkg: 527 D 	home = /usr/bin [virtualenv/create/pyenv_cfg.py:34]
.pkg: 527 D 	implementation = CPython [virtualenv/create/pyenv_cfg.py:34]
.pkg: 527 D 	version_info = 3.10.10.final.0 [virtualenv/create/pyenv_cfg.py:34]
.pkg: 527 D 	virtualenv = 20.21.0 [virtualenv/create/pyenv_cfg.py:34]
.pkg: 527 D 	include-system-site-packages = false [virtualenv/create/pyenv_cfg.py:34]
.pkg: 528 D 	base-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
.pkg: 528 D 	base-exec-prefix = /usr [virtualenv/create/pyenv_cfg.py:34]
.pkg: 528 D 	base-executable = /usr/bin/python3.10 [virtualenv/create/pyenv_cfg.py:34]
.pkg: 528 W install_requires> python -I -m pip install 'setuptools>=40.8.0' wheel [tox/tox_env/api.py:428]
Requirement already satisfied: setuptools>=40.8.0 in ./.tox/.pkg/lib/python3.10/site-packages (67.5.1)
Requirement already satisfied: wheel in ./.tox/.pkg/lib/python3.10/site-packages (0.38.4)
.pkg: 1241 I exit 0 (0.71 seconds) /home/dharding/code/tox_test> python -I -m pip install 'setuptools>=40.8.0' wheel pid=335223 [tox/execute/api.py:275]
.pkg: 1250 W _optional_hooks> python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ [tox/tox_env/api.py:428]
Backend: run command _optional_hooks with args {}
Backend: Wrote response {'return': {'get_requires_for_build_sdist': True, 'prepare_metadata_for_build_wheel': True, 'get_requires_for_build_wheel': True, 'build_editable': True, 'get_requires_for_build_editable': True, 'prepare_metadata_for_build_editable': True}} to /tmp/pep517__optional_hooks-8p38cl4r.json
.pkg: 1394 I exit None (0.14 seconds) /home/dharding/code/tox_test> python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ pid=335240 [tox/execute/api.py:275]
.pkg: 1395 W get_requires_for_build_sdist> python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ [tox/tox_env/api.py:428]
Backend: run command get_requires_for_build_sdist with args {'config_settings': None}
running egg_info
writing tox_test.egg-info/PKG-INFO
writing dependency_links to tox_test.egg-info/dependency_links.txt
writing top-level names to tox_test.egg-info/top_level.txt
reading manifest file 'tox_test.egg-info/SOURCES.txt'
writing manifest file 'tox_test.egg-info/SOURCES.txt'
Backend: Wrote response {'return': []} to /tmp/pep517_get_requires_for_build_sdist-nqy3wvr1.json
.pkg: 1417 I exit None (0.02 seconds) /home/dharding/code/tox_test> python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ pid=335240 [tox/execute/api.py:275]
.pkg: 1418 W prepare_metadata_for_build_wheel> python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ [tox/tox_env/api.py:428]
Backend: run command prepare_metadata_for_build_wheel with args {'metadata_directory': '/home/dharding/code/tox_test/.tox/.pkg/.meta', 'config_settings': {'--build-option': []}}
running dist_info
creating /home/dharding/code/tox_test/.tox/.pkg/.meta/tox_test.egg-info
writing /home/dharding/code/tox_test/.tox/.pkg/.meta/tox_test.egg-info/PKG-INFO
writing dependency_links to /home/dharding/code/tox_test/.tox/.pkg/.meta/tox_test.egg-info/dependency_links.txt
writing top-level names to /home/dharding/code/tox_test/.tox/.pkg/.meta/tox_test.egg-info/top_level.txt
writing manifest file '/home/dharding/code/tox_test/.tox/.pkg/.meta/tox_test.egg-info/SOURCES.txt'
reading manifest file '/home/dharding/code/tox_test/.tox/.pkg/.meta/tox_test.egg-info/SOURCES.txt'
writing manifest file '/home/dharding/code/tox_test/.tox/.pkg/.meta/tox_test.egg-info/SOURCES.txt'
creating '/home/dharding/code/tox_test/.tox/.pkg/.meta/tox_test-0.0.0.dist-info'
Backend: Wrote response {'return': 'tox_test-0.0.0.dist-info'} to /tmp/pep517_prepare_metadata_for_build_wheel-x1dyybxp.json
.pkg: 1493 I exit None (0.07 seconds) /home/dharding/code/tox_test> python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ pid=335240 [tox/execute/api.py:275]
.pkg: 1494 W build_sdist> python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ [tox/tox_env/api.py:428]
Backend: run command build_sdist with args {'sdist_directory': '/home/dharding/code/tox_test/.tox/.pkg/dist', 'config_settings': None}
running sdist
running egg_info
writing tox_test.egg-info/PKG-INFO
writing dependency_links to tox_test.egg-info/dependency_links.txt
writing top-level names to tox_test.egg-info/top_level.txt
reading manifest file 'tox_test.egg-info/SOURCES.txt'
writing manifest file 'tox_test.egg-info/SOURCES.txt'
running check
creating tox_test-0.0.0
creating tox_test-0.0.0/tox_test.egg-info
copying files to tox_test-0.0.0...
copying README -> tox_test-0.0.0
copying setup.py -> tox_test-0.0.0
copying tox_test.egg-info/PKG-INFO -> tox_test-0.0.0/tox_test.egg-info
copying tox_test.egg-info/SOURCES.txt -> tox_test-0.0.0/tox_test.egg-info
copying tox_test.egg-info/dependency_links.txt -> tox_test-0.0.0/tox_test.egg-info
copying tox_test.egg-info/top_level.txt -> tox_test-0.0.0/tox_test.egg-info
Writing tox_test-0.0.0/setup.cfg
Creating tar archive
removing 'tox_test-0.0.0' (and everything under it)
Backend: Wrote response {'return': 'tox_test-0.0.0.tar.gz'} to /tmp/pep517_build_sdist-lr1o6egx.json
.pkg: 1509 I exit None (0.02 seconds) /home/dharding/code/tox_test> python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ pid=335240 [tox/execute/api.py:275]
.pkg: 1510 D package .tmp/package/14/tox_test-0.0.0.tar.gz links to .pkg/dist/tox_test-0.0.0.tar.gz (/home/dharding/code/tox_test/.tox) [tox/util/file_view.py:36]
testenv: 1510 W install_package> python -I -m pip install --force-reinstall --no-deps /home/dharding/code/tox_test/.tox/.tmp/package/14/tox_test-0.0.0.tar.gz [tox/tox_env/api.py:428]
Processing ./.tox/.tmp/package/14/tox_test-0.0.0.tar.gz
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: tox-test
  Building wheel for tox-test (setup.py): started
  Building wheel for tox-test (setup.py): finished with status 'done'
  Created wheel for tox-test: filename=tox_test-0.0.0-py3-none-any.whl size=936 sha256=4898c208cd2bab225a2fc7533086be91487e6dedafeaf0d694e2238fc7936249
  Stored in directory: /home/dharding/.cache/pip/wheels/18/8d/22/ffc3acf6c491acdf99a09c8acb1a81a7890f26f2d3b69dd6fd
Successfully built tox-test
Installing collected packages: tox-test
Successfully installed tox-test-0.0.0
testenv: 2616 I exit 0 (1.11 seconds) /home/dharding/code/tox_test> python -I -m pip install --force-reinstall --no-deps /home/dharding/code/tox_test/.tox/.tmp/package/14/tox_test-0.0.0.tar.gz pid=335288 [tox/execute/api.py:275]
testenv: 2617 W commands[0]> python -c 'import os; import sys; sys.exit(not '"'"'BAR'"'"' in os.environ)' [tox/tox_env/api.py:428]
testenv: 2626 C exit 1 (0.01 seconds) /home/dharding/code/tox_test> python -c 'import os; import sys; sys.exit(not '"'"'BAR'"'"' in os.environ)' pid=335316 [tox/execute/api.py:275]
.pkg: 2626 W _exit> python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ [tox/tox_env/api.py:428]
Backend: run command _exit with args {}
Backend: Wrote response {'return': 0} to /tmp/pep517__exit-378ndfd1.json
.pkg: 2628 I exit None (0.00 seconds) /home/dharding/code/tox_test> python /home/dharding/code/opensource/tox/.tox/dev/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__ pid=335240 [tox/execute/api.py:275]
.pkg: 2641 D delete package /home/dharding/code/tox_test/.tox/.tmp/package/14/tox_test-0.0.0.tar.gz [tox/tox_env/python/virtual_env/package/pyproject.py:179]
  testenv: FAIL code 1 (2.37=setup[2.36]+cmd[0.01] seconds)
  evaluation failed :( (2.54 seconds)

Minimal example

If possible, provide a minimal reproducer for the issue:

The following tox.ini works on tox 3.x (the BAR environment variable is present as expected) but fails on tox 4.x (the BAR environment variable should be inherited from [testenv:base]set_env but is absent):

[tox]
envlist =
    main

[testenv:base]
set_env =
    BAR = 1

[testenv:main]
set_env =
    {[testenv:base]set_env}
    FOO = {env:FOO2:foo}
commands = python -c "import os; import sys; sys.exit(not 'BAR' in os.environ)"

The issue is that SetEnv.__iter__() gets called recursively, which effectively causes some of the keys to be lost. The first time it is called via VirtualEnv.session(), which ends up invoking ToxEnv.environment_variables(). At this point, the SetEnv._raw dict has one key, FOO, and SetEnv._needs_replacement is ['{[testenv:base]set_env}']. ToxEnv.environment_variables() calls the SetEnv.load() method on each key that SetEnv.__iter__() yields, so once SetEnv.__iter__() yields FOO (as a result of this line), ToxEnv.environment_variables() calls set_env.load('FOO') (see here). In the process of resolving the value for FOO, the replace_env() function ends up being called. This in turn calls SetEnv.__contains__() which in turn calls SetEnv.__iter__() a second time. Because the key being tested (in this case FOO2) is not present, the second call to SetEnv.__iter__() will iterate over all the keys, including expanding the entries from the self._needs_replacement attribute. Those keys will be stored in the self._raw attribute, but since the the first call to SetEnv.__iter__() has already made a copy of the self._raw attribute to protect against concurrent mutation, it cannot yield them. Furthermore, once it is done yielding from its copy of the self._raw keys, the first call finds that self._needs_replacement is empty, because the second call to SetEnv.__iter__() has already processed all of those entries. Thus the first SetEnv.__iter__() call never yields the keys from self._needs_replacement which were expanded by the second .__iter__() call, and from the perspective of the ToxEnv.environment_variables() call, they are lost.

Below is an annotated call stack from when the problem occurs:
  File "/usr/lib/python3.10/threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 83, in _worker
    work_item.run()
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/dharding/code/opensource/tox/src/tox/session/cmd/run/common.py", line 304, in _run
    return run_one(
  File "/home/dharding/code/opensource/tox/src/tox/session/cmd/run/single.py", line 34, in run_one
    skipped, code, outcomes = _evaluate(tox_env, no_test)
  File "/home/dharding/code/opensource/tox/src/tox/session/cmd/run/single.py", line 45, in _evaluate
    tox_env.setup()
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/api.py", line 249, in setup
    self._setup_env()
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/python/runner.py", line 106, in _setup_env
    super()._setup_env()
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/python/api.py", line 198, in _setup_env
    self.ensure_python_env()
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/python/api.py", line 202, in ensure_python_env
    conf = self.python_cache()
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/python/virtual_env/api.py", line 77, in python_cache
    base = super().python_cache()
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/python/api.py", line 240, in python_cache
    "version_info": list(self.base_python.version_info),
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/python/api.py", line 250, in base_python
    self._base_python = self._get_python(base_pythons)
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/python/virtual_env/api.py", line 134, in _get_python
    interpreter = self.creator.interpreter
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/python/virtual_env/api.py", line 126, in creator
    return self.session.creator
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/python/virtual_env/api.py", line 106, in session
    env = self.virtualenv_env_vars()
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/python/virtual_env/api.py", line 111, in virtualenv_env_vars
    env = self.environment_variables.copy()
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/python/virtual_env/api.py", line 166, in environment_variables
    environment_variables = super().environment_variables
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/runner.py", line 191, in environment_variables
    environment_variables = super().environment_variables
  File "/home/dharding/code/opensource/tox/src/tox/tox_env/api.py", line 341, in environment_variables <-- The first loop over SetEnv.__iter__() is happening here.
    result[key] = set_env.load(key)
  File "/home/dharding/code/opensource/tox/src/tox/config/set_env.py", line 78, in load
    result = self._replacer(raw, args)  # apply any replace options
  File "/home/dharding/code/opensource/tox/src/tox/config/loader/ini/__init__.py", line 77, in replacer
    replaced = replace(conf, self, raw_, args_)  # do replacements
  File "/home/dharding/code/opensource/tox/src/tox/config/loader/ini/replace.py", line 57, in replace
    return Replacer(conf, loader, conf_args=args, depth=depth).join(find_replace_expr(value))
  File "/home/dharding/code/opensource/tox/src/tox/config/loader/ini/replace.py", line 185, in join
    return "".join(self(value))
  File "/home/dharding/code/opensource/tox/src/tox/config/loader/ini/replace.py", line 182, in __call__
    return [self._replace_match(me) if isinstance(me, MatchExpression) else str(me) for me in value]
  File "/home/dharding/code/opensource/tox/src/tox/config/loader/ini/replace.py", line 182, in <listcomp>
    return [self._replace_match(me) if isinstance(me, MatchExpression) else str(me) for me in value]
  File "/home/dharding/code/opensource/tox/src/tox/config/loader/ini/replace.py", line 197, in _replace_match
    replace_value = replace_env(self.conf, args, conf_args)
  File "/home/dharding/code/opensource/tox/src/tox/config/loader/ini/replace.py", line 331, in replace_env
    if key in set_env:
  File "/home/dharding/code/opensource/tox/src/tox/config/set_env.py", line 86, in __contains__
    return isinstance(item, str) and item in self.__iter__() <-- The second call to SetEnv.__iter__()  happens here
@gaborbernat
Copy link
Member

PR welcome. 👍

@living180
Copy link
Contributor Author

So, question which would guide a potential PR: I've observed another behavior discrepancy between tox 3 and tox 4, and am not sure if I should try to restore the tox 3 behavior or maintain the behavior in tox 4 (I could not find anything in either the tox 3 or tox 4 documentation that specifies the expected behavior).

In tox 3, the order of section substitution relative to explicit keys is significant:

[tox]
envlist =
    main

[testenv:base]
setenv =
    FOO = 2

[testenv:main]
setenv =
    FOO = 3
    {[testenv:base]setenv}

will set the environment variable FOO to 2 (the section substitution will take precedence over the explicit key that came before it), while

[tox]
envlist =
    main

[testenv:base]
setenv =
    FOO = 2

[testenv:main]
setenv =
    {[testenv:base]setenv}
    FOO = 3

will set the environment variable FOO to 3 (the explicit key, coming after the section substitution, will take precedence).

However, in tox 4, for both of the examples, FOO is 3 (section substitution never takes precedence over explicit keys).

Any input regarding what the desired behavior should be is welcome.

@gaborbernat
Copy link
Member

This is expected 🤔 though can be convinced that last found key should win over first found 🤷

@living180
Copy link
Contributor Author

So another question, if you don't mind: commit 4cf619a introduced what looks to me like a level of indirection using concurrent.futures.Future as well as a new .build() method of Loader / IniLoader. I'm not sure of the purpose of this mechanism, or what problem it is trying to solve. Is it possible to shed any light on this?

@gaborbernat
Copy link
Member

The core idea is to make configuration definition and manifestation lazy. So tox is now faster because it only does work when it must.

mr-mixas added a commit to mr-mixas/Nested-Diff.py that referenced this issue May 12, 2023
setenv defined explicitly until regression fixed in tox:
* tox-dev/tox#2972
* tox-dev/tox#2872
mr-mixas added a commit to mr-mixas/Nested-Diff.py that referenced this issue May 12, 2023
setenv defined explicitly until regression fixed in tox:
* tox-dev/tox#2972
* tox-dev/tox#2872
mr-mixas added a commit to mr-mixas/Nested-Diff.py that referenced this issue May 12, 2023
setenv defined explicitly until regression fixed in tox:
* tox-dev/tox#2972
* tox-dev/tox#2872
@living180
Copy link
Contributor Author

I had to put this issue on the back burner due to other responsibilities, but today I had a bit of time to look at it again. So it looks like tox is making use of futures in tox.session.cmd.run.common._queue_and_wait(), which makes sense to me. However, the usage in tox.config.loader.api I'm still not sure about. Loader.load() creates a Future() manually which the Python docs say should not be done normally, and no executor is involved in that code. Was this some preparatory work that was never finished? As far as I understand this bit of code, it isn't actually doing anything in a lazy manner.

@gaborbernat
Copy link
Member

You could try to remove it 🤔 in a PR.

@gaborbernat gaborbernat added the help:wanted Issues that have been acknowledged, a solution determined and a PR might likely be accepted. label Jun 16, 2023
@cladmi
Copy link

cladmi commented Jun 26, 2023

I also faced the same issue.
One note, this only affects the execution environment and not the {env:} substitution.

The substitution done inside tox file works as expected, but the environment of commands is not set properly.

[tox]
envlist = py3,py3_working

[testenv]
skip_install = true
setenv =
    ROOTDIR = {toxinidir}
commands =
    python -c 'print("ROOTDIR={env:ROOTDIR}")'
    python -c 'import os; print(os.environ["ROOTDIR"])'

[testenv:py3]
setenv =
    {[testenv]setenv}
    EXTRA_VAR = --rootdir={env:TOX_ENV_NAME}

[testenv:py3_working]
setenv =
    {[testenv]setenv}
    EXTRA_VAR = --rootdir={toxinidir}
tox
py3: commands[0]> python -c 'print("ROOTDIR=/tmp/check_tox_env")'
ROOTDIR=/tmp/check_tox_env
py3: commands[1]> python -c 'import os; print(os.environ["ROOTDIR"])'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'ROOTDIR'
py3: exit 1 (0.02 seconds) /tmp/check_tox_env> python -c 'import os; print(os.environ["ROOTDIR"])' pid=711194
py3: FAIL ✖ in 0.1 seconds
py3_working: commands[0]> python -c 'print("ROOTDIR=/tmp/check_tox_env")'
ROOTDIR=/tmp/check_tox_env
py3_working: commands[1]> python -c 'import os; print(os.environ["ROOTDIR"])'
/tmp/check_tox_env
  py3: FAIL code 1 (0.10=setup[0.05]+cmd[0.02,0.02] seconds)
  py3_working: OK (0.06=setup[0.01]+cmd[0.02,0.02] seconds)
  evaluation failed :( (0.37 seconds)

A workaround is to move all the variables setting that I want exported out of the common environment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help:wanted Issues that have been acknowledged, a solution determined and a PR might likely be accepted.
Projects
None yet
Development

No branches or pull requests

3 participants