-
Notifications
You must be signed in to change notification settings - Fork 22k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PyTorch trunk is frequently broken #128180
Comments
Another doc-only change that somehow failed rebuilding docker containers #128136 |
All these unrelated failures are linked one way or another to the ongoing CUDA 12.4, CUDNN upgrade. So an approach could be to figure out how to roll out similar upgraded in a safer manner. cc @atalman |
If we zoom into doc-only changes, there is a TD item to skip CI jobs if the PR only updates comments / docs. However, that capacity is not yet being built. cc @clee2000 |
The failures with cudnn.so are related to the fact that we have 2 changes:
These 2 changes are landed one after another. But people who not rebased are missing pytorch/pytorch changes. The docker changes will be picked up automatically. Perhaps we should open non blocking CI SEV when doing such updates ? Please note this is major update - we have not updated cudnn major version since 2021 at least. |
I've noticed a higher influx of red on my PRs lately but had no idea that it was correlated with a cudnn update. If we know that these big updates could disrupt CI for all devs, it would be good to announce that risk more vocally so I could know what to expect. |
@atalman , which is the pytorch/pytorch commit that people need to rebase past to not get these spurious cuda failures? Let's mention that as part of the mitigation steps in the sev |
Are the libcudnn.so failures (marked as unstable/flaky in @malfet's screenshot) also fixed by the pytorch/pytorch rebase, or are they a symptom of a separate issue? |
Looking through the logs, we started seeing the |
I understand that everything can be solved by a rebase and retry, I've filed this issue to highlight that this expectation to rebase seems to become more and more common. |
@malfet We will start working on this : pytorch/builder#1849 - Moving docker workflow to pytorch/pytorch and fetching correct docker image should take care of the errors similar to "libcudnn.so.9" |
Another example is here: #127456 ![]() |
from triage review: keeping hi pri to track status, will likely be an ongoing issue since there are usually new reasons for breakages. Andrey: cuda docker builds run in two steps, can move the docker builds to pytorch/pytorch to avoid some cuda CI errors. But there are multiple different failures going on here. |
@atalman: To break this down into smaller actionable items before closing the issue |
One more example from PR #128222 |
Are there AIs here that still need to be done? I've lost context since the issue is a bit old |
🐛 Describe the bug
For example looking at #128123 (that adds a docstring to the utils and therefore could not cause any failures other than lint) DrCI found 2 unstable job, 1 flaky and one new failure in inductor tests:
Not sure how new developer who want to make a small contribution to PyTorch support to navigate thru those multiple layers of dilapidation...
Versions
CI
cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @seemethere @pytorch/pytorch-dev-infra
The text was updated successfully, but these errors were encountered: