Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

yolov5 model #7879

Closed
1 task done
jafar0892 opened this issue May 18, 2022 · 2 comments
Closed
1 task done

yolov5 model #7879

jafar0892 opened this issue May 18, 2022 · 2 comments
Labels
question Further information is requested Stale

Comments

@jafar0892
Copy link

Search before asking

Question

Hi, i would like train a yolov5 modell with large batch size, but unfortunately i am getting error that CUDA is out of memory. is it any solution to train it with large batch size to get better result and not getting the error ?

Additional

No response

@jafar0892 jafar0892 added the question Further information is requested label May 18, 2022
@glenn-jocher
Copy link
Member

glenn-jocher commented May 18, 2022

@jafar0892 👋 Hello! Thanks for asking about CUDA memory issues. YOLOv5 🚀 can be trained on CPU, single-GPU, or multi-GPU. When training on GPU it is important to keep your batch-size small enough that you do not use all of your GPU memory, otherwise you will see a CUDA Out Of Memory (OOM) Error and your training will crash. You can observe your CUDA memory utilization using either the nvidia-smi command or by viewing your console output:

Screenshot 2021-05-28 at 12 19 51

CUDA Out of Memory Solutions

If you encounter a CUDA OOM error, the steps you can take to reduce your memory usage are:

  • Reduce --batch-size
  • Reduce --img-size
  • Reduce model size, i.e. from YOLOv5x -> YOLOv5l -> YOLOv5m -> YOLOv5s > YOLOv5n
  • Train with multi-GPU at the same --batch-size
  • Upgrade your hardware to a larger GPU
  • Train on free GPU backends with up to 16GB of CUDA memory: Open In Colab Open In Kaggle

AutoBatch

You can use YOLOv5 AutoBatch (NEW) to find the best batch size for your training by passing --batch-size -1. AutoBatch will solve for a 90% CUDA memory-utilization batch-size given your training settings. AutoBatch is experimental, and only works for Single-GPU training. It may not work on all systems, and is not recommended for production use.

Screenshot 2021-11-06 at 12 31 10

Good luck 🍀 and let us know if you have any other questions!

@github-actions
Copy link
Contributor

github-actions bot commented Jun 18, 2022

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

Access additional Ultralytics ⚡ resources:

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested Stale
Projects
None yet
Development

No branches or pull requests

2 participants