-
Notifications
You must be signed in to change notification settings - Fork 524
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CUDA out of memory after some epochs #199
Comments
Hi,
and line:
not sure why the batch size is doubled during validation, but that solved the issue for me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm training on an EC2 instance with T4 GPU and 16GB of memory.
I'm using a batch size of 2 and image size of 960, however after 3 epochs the script is killed because GPU is out of memory. How can I overcome this without reducing my batch size to 1?
Thanks for the help.
The text was updated successfully, but these errors were encountered: