Allow substitution from the current config section

[jayvdb]

The substitution engine allows access to any key from another section using {[section_name]key_name}, however afaics there isnt a way to do the same for the current section. It is possible to access predefined substitution names which are often the same as key, such as {commands}, however adhoc keys in the current section cant be accessed.

Another odd side effect of this is that {[testenv:other]deps} works, but it is not possible to get the deps of the current section without repeating the current section name. e.g.

[testenv:current]
deps = distro
commands = pip install {[testenv:current]deps}

A nice conceptual syntax would be {[.]key_name}, however . is a valid section name. Personally I think that . is a dumb section name, whereas foo.bar might reasonably be in use. So making . by itself an illegal section name (or as the first char of the section name), especially if given a nice error, makes sense. I am wondering why . is legal in _FACTOR_LINE_PATTERN. There doesnt appear to be any special meaning for . so perhaps there is a good reason for it to be valid in testenv names.

If . is not acceptable, I propose some others

1. {[:]key_name}, {[testenv::]commands} and {[:]commands} does currently work. Despite being legal, section name : is more obviously problematic than .. There is a bit of syntax overlap as {:} refers to os.pathsep, however { is invalid within any {..} syntax, and there is no intersection of "ini format" and os.pathsep , so it doesnt seem using {[:]..} is going to be confusing. More importantly, testenv:: is an error in virtualenv:

virtualenv: error: argument dest: destination ':' must not contain the path separator (:) as this would break the activation scripts

2. {[]key_name}, which currently internally becomes {[:]key_name}! i.e. {[]commands} does currently work, and it gets commands from section :. However, {[]foo} is sort-of over-loading [], because [] in commands means {posargs}.

3. {[-]key_name}, as - and testenv:- is an even more stupid section name. {[-]commands]} and {[testenv:-]commands]} do work, and envlist = - does run testenv:- however it is obviously problematic due to factoring, and should probably be explicitly an error because it is very likely this will break in odd ways, if not now it could easily break due to very minor implementation changes in the future.

4. {[!]key_name}, as ! is "reserved" for factor negating, and is invalid in testenv names. {[!]commands]} and {[testenv:!]commands]} do work, and envlist = ! does run testenv:! however it is obviously problematic due to factoring, and should probably be explicitly an error because it is very likely this will break in odd ways, if not now it could easily break due to very minor implementation changes in the future. However using ! to refer to the current section feels illogical.

5. {[#]key_name}, as # is sort-of reserved as a comment char in some contexts, envlist=# doesnt work, and [#] and [testenv:#] appears to cause iniformat parsing problems. {[#]foo} doesnt immediately seem intuitive, but # has a history of referring to sections/anchors, and without a suffix isnt pointing to a different section, so it isnt much of a stretch to use it to refer to the current section.

6. {[,]key_name}, as testenv:, is also invalid, as it resolves to an empty string, and {[,]commands} is also invalid (max-recursion!), and so is {[testenv:,]commands} ("substitution key '[testenv' not found" :/ ) , but {[,]key_name} is ugly - it doesnt make any sense.

Of the above, (2) {[]key_name} is my preference, because it currently works, but works in an unexpected way, so giving it a new sane meaning also fixes a bug at the same time.

I also like (5) {[#]key_name} because it is the only symbol which is completely unusable currently in any related context, and has a very strong meaning (comment) in most contexts and is quite hard to make it valid in any context, e.g. file paths (#763.

* is another symbol which is illegal in some contexts as it is used for globbing. Globbing appears to be occurring in deps, passenv, allowlist_externals, TOX_SKIP_ENV, possibly also TOXENV.
In addition, using {[*]foo} to refer to the current section is even more illogical than using ,. Also [testenv:*] and "[.*] pattern to denote positional arguments with defaults" are mentioned in docs config.html .

| is used only in the v3.20+ file|. It doesnt seem appropriate to use {[|]key_name}, and I suspect | will grow more meaning over time.

There are many other symbols which could be used instead of the above, but they will very likely be currently valid in section names, albeit probably not usable in envlist. e.g. within ascii, std keyboard +, ?, ^, %, $, @, & , and backtick. It appears ( and ) are also unused so far, however those and =, ~, < and > are commonly used in deps pip syntax, so is ; and @ (and , and !). In any case, none of them seem obviously desirable for referring to the current section.

Semantically, "{[]foo}" should refer to the section where the resolution is occurring, not where it is defined. (because it is already possibly to explicitly refer to the section where it is defined) i.e. with

[testenv:base]
deps = distro
commands = pip install {[]deps}

[testenv:other]
deps = foobar
commands = {[testenv:base]commands}

tox -e other should install foobar, not distro.

This would mean that {[]deps} provides the same result as the unimplemented {packages}.

[jayvdb]

This would mean that {[]deps} provides the same result as the unimplemented {packages}.

Note my PR doesnt even contemplate how to represent complex values like {deps} sanely in the current context, similar to how {[other]deps} could be incompatible if used in setenv or commands. I havent found many open issues about coercing values into different variable types, so it doesnt seem like there is much need for that.

[gaborbernat]

This is the easiest to read, understand and reason about it:

[testenv:current]
deps = distro
commands = pip install {[testenv:current]deps}

All other forms require the user to know the internal syntax. We defined how to resolve it. I'm personally -0.5 introducing a syntax just for these because:

• in the spirit of python, explicit is better than implicit,
• users already find our syntax hard to learn, and I'd not make that even harder by adding more to it.

I find needing to refer to the current section rarely enough to qualify the reference fully is not cumbersome.

PS. Your example is not valid because you should not need to pip install deps manually; tox already takes care of it for you. So at the very least, I want some real world examples where the status quo is bad.

[jayvdb]

The examples I gave are contrived, only for the purpose of showing inheritance. I should have used echo, and could have used install_command instead of commands, or any other keys.

One problem with explicit literal "current" is

[testenv]
deps = distro

[testenv:current]
commands = echo {[testenv:current]deps}

tox.exception.ConfigError: ConfigError: substitution key '[testenv:current]deps' not found

That might be solvable by adding inheritance/fallback-sections to the "other section" resolution. Happy to explore that if you see that as a useful approach. It does seem bug-ish that "other section" resolution doesnt respect testenv inheritance/fallback-sections, and there is no backwards incompatibility possible for implementing it.

echo there could be instead a command which checks the venv is correctly populated. There is at least one open issue about the potential for setup.py to have corrupted the venv with packages incompatible with deps - rather than tox fixing that directly, which would be a perf hit, IMO a reasonable approach is to let people use tools to detect this problem, which would be only needed for complex use-cases. In any case, why people might want to do something is IMO beside the point - tox benefits from providing generic methods to achieve unusual goals. I don't mind the push back, needing to show reasonable use-cases, and can provide more open issues which can be solved directly or indirectly by same-section-with-testenv-inheritance value substitution.

[gaborbernat]

That might be solvable by adding inheritance/fallback-sections to the "other section" resolution

Yeah, this one. I think I have this functionality work on the rewrite branch.

There is at least one open issue about the potential for setup.py to have corrupted the venv with packages incompatible with deps

I think we can do this without significant performance hits, once we optimize things a bit.

[tucked]

Does {[testenv:{envname}]deps} help here at all?

edit: related: #1822

[gaborbernat]

This now has been implemented in the rewrite branch. I don't think we plan to support this feature on tox 3.

[gaborbernat]

Closing as has been fixed on the rewrite branch.