-
-
Notifications
You must be signed in to change notification settings - Fork 512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
option to fail fast #578
Comments
for tox running everything is expectedbehaviour |
That seems like expected behaviour to me. If e.g. tests with python 2 fail, I'd still like to run tests with python 3. |
Hi @jtiai as already said: this is how it's supposed to be :) You could get to the wanted behaviour by doing something like this on the command line (same semantics on POSIX and Windows):
If any of those runs fail the rest of the chain will not be executed anymore. On CI systems you usually have the concept of stages that are executed serially and if one stage fails the whole plan fails and nothing else is executed. So there you would create a stage for each of the tox environments that depend on each other. |
I got the impression that tox stops the execution from the docs: So maybe it's just matter of clarify in the documentation that only current environment execution is aborted, but rest of the environments defined will be executed regardless state of the others. |
@jtiai Do you have any suggestion on how this could be clarified? It already says "will abort execution of commands for that environment.". |
Last paragraph states:
So that's where I got impression that tox stops on error. Not sure about wording but that part should somehow emphasize the fact that only current environment execution is stopped and if there are more defined, either in envlist configuration or via -e switch from the cli rest will be executed. |
I think we only need to point out that that abort is active inside the commands 👍 not for the environments 👍 |
Take a look at ignore_outcome's doc as well. Now it says:
Really expecting tox to fail by default after some testenv fails. |
o.k. that surprises me, but I get now where you are coming from, so we definitely need to make the docs clearer in those areas. Thanks! |
Hi @apomelov, this also contains the term "testenv" which means that it only means that the testenv won't fail, but I get it. I think what is missing is a bit of a high level overview that is describing a tox run and everything that is involved and properly introduces the terminology. |
And reading it again I realize the misunderstanding is a different one here ... sorry @apomelov. What this means is that the whole tox run will not be regarded as a failure, if the testenv fails if that flag is set. It will still run all the environments. So this should be explicitly added there I guess. |
Next thought is obviously to introduce a flag that tox should stop running if one testenv fails, but I see trouble that way (e.g. what is detox supposed to do, which will end up in core at some point). |
well, quite simple, once one fails, don't schedule new ones to get started |
This falls down with detox though. I still think that tox should just run all the envs you tell it to run. If you want to run them staged then address them like I mentioned above. |
thats precisely why i used the word schedule instead of run, i didn't want to suggest to cancel already running ones, but i did want to suggest to simply stop scheduling additional executions after the first failure |
This would be really useful for running tox within git-bisect |
I would find this option useful as well. I'd like my linter, unit tests, and end-to-end tests in different environments, and there should be an option to stop early if the linter or unit tests fail to avoid running the long e2e tests. E.g. something like this: |
For development perspective I would find this very useful because I don't want to scroll lots of screens in order to see what failed. I just want to get to the failure as quick as possible, fix it and run again. I can understand that for CI the desire may be different but if this feature would be controlable using both tox.ini entry and a TOX_ variable it would be ideal, as I could just define the variable for my development machine and not have to touch the codebase. |
right now i'm dong a big refactor on a big python project on the one hand, this leaves a lot of time for contemplation: why am i doing this? why am i not doing something i get paid for? why didn't i listen to my parents and learn something sensible like goat herding? on the other hand it would be quite useful to get this done quickly and have less existential angst. |
@igalic if you're refactoring you don't need to run the entire test suite all the time. You can use the |
Hi @igalic, does that mean that in order to find out what is going wrong, you always have to run all toxenvs? Otherwise I would also go with the suggestion from Bernát or even activate the toxenv and run specific tests there until you are happy and only then run the whole suite on all envs to see what broke (Ideally in parallel on your CI or with detox locally to speed things up).
I think that CI should be responsible for this kind of thing (e.g. in the case of Travis via build stages, which call the toxenvs in the order they are needed, failing whenever a stage fails). |
Sorry my bad. the milestone is about the documentation part of this sigh of relief. |
indeed, running only one of the envs greatly cut down on waiting and scrolling time, and i was able to make some actual progress! thank you all for the advice! |
I would love to see this feature as well, we have thousands of tests and it's quite hard to scroll up in the list to find the previous failures. |
@jdahlin those thousand test running are handled by your test runner, not by tox; so the test runner provides the availability to stop, not tox. All we can do is that if py37 failed we don't start py36 too... but that's tox environment count, not test count. |
Any chance of making this happen? I got others complaining about the same issues. With Travis resource limitations we do often want to group multiple environments inside a single job. Using the workaround of calling tox multiple times for each environment works would required an almost complete rewrite of some complex travis.yaml files. See ansible/molecule#2257 (review) where one colleague opposed consolidation for the same reasons. Maybe this does not appear as a big issue for small projects, but if you end-up having 27 environments like we have on molecule, you realize that this bug becomes a real problem. |
Chances are that yes, within tox 4 I would say, but not likely to happen before end of year. |
+1 for this feature. We utilize Tox in our CI/CD pipeline and it would greatly shorten testing time if there was an option for Tox to fail fast on error. |
I would also like to pile on to this. I was quite surprised that this wasn't an available option. I agree that it shouldn't be the default, just as it's not the default for pytest, but it's a useful option. |
Hi, any ETA on this? |
Still caught up in trying to stabilise virtualenv, I'd say no chance before end of year 👍for this to happen with tox 4. |
Do you need any help on implementing this? I really need this ability for my project and would love to give a hand on it :) |
If someone wants to implement it against current master I can help review it and land it 🤷♂️ |
If it needs to only stop some envs if one envs fails Sample Situation :
Maybe this approach helps 😃 [tox]
envlist = {py27,py35}-batch1,{pypy2,pypy3}-batch2
[testenv:batch1]
stoponfail = True
[testenv:batch2]
stoponfail = True |
Until this will be implemented, I'm using for loop for it like: for env in `tox -l`; do echo $env; tox -e $env || break; done |
Any word on this issue ? For my use-case, there are just some Just to add my two cents, I think the following implementation would be great (where [tox]
requires =
tox>=4
env_list =
format
py312
[testenv:format]
description = run formatters on src code
stop_on_fail = true
deps =
black
commands =
black src
[testenv]
description = run test code
deps =
pytest
commands =
pytest test Are there any objections or known technical pain-points for such an implementation ? Or is it more a matter of bandwidth at the moment ? |
Yes, PR is welcome. We do not plan to do this ourselves. |
If having configure multiple environments and running them with plain
tox
-command if environment in the middle fails, rest of the environments are run.Expected that if one environment fails, rest aren't executed.
Sample tox.ini:
pip list:
The text was updated successfully, but these errors were encountered: