-
Notifications
You must be signed in to change notification settings - Fork 269
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RuntimeError When Enabling Accuracy Checks in DALLE2_pytorch Training on GPU. #2241
Comments
I can confirm that this can be reproduced in the docker environment. |
@xuzhao9 The problem also occurs on the previous version of TorchBench(ghcr.io/pytorch/torchbench:dev20230619). It looks like it is from the first time DALLE2 was included in TorchBench. I'm not sure if we can fix it on our side or from the upstream repo since we have limited control over the model's init.py. I'll have a try. |
We are dropping DALLE2_pytorch because it does not support numpy 2.0: #2311 |
Issue Description
I encounter a RuntimeError related to gradient computation when enabling accuracy checks during the training of DALLE2_pytorch in a GPU docker environment. The training runs without issues when the
--accuracy
flag is not used.Steps to Reproduce
python install.py DALLE2_pytorch
python run.py DALLE2_pytorch -d cuda -t train --accuracy
Expected Behavior
The training process should run without errors and perform accuracy checks without causing runtime errors.
Actual Behavior
The script executes successfully without the --accuracy flag.
However, when the accuracy check is enabled, it fails with the following error message:
Additional Context
PyTorch version: 2.2.2
CUDA version: 12.4.0.041
The text was updated successfully, but these errors were encountered: