-
Notifications
You must be signed in to change notification settings - Fork 982
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Training speed in bf16 mode is slow. #660
Labels
bug
Something isn't working
Comments
Did your comparison fp16 model use |
yes, my comparison fp 16 model is using zero-1. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I trained a model from scratch on RTX 3090 cards (
torch.cuda.is_bf16_supported()
returns True) in bfloat16 mode, but the training speed turns out to be 1/3 of the speed in fp16 mode (I use the "elapsed time per iteration (ms)" log field for comparision )To Reproduce
I borrowed the following settings from the
configs/small_bf16.yml
file for the bf16 mode, the other setting are same to the fp16 mode.Expected behavior
According to huggingface/transformers#14608,
Proposed solution
If you have an idea for how we can fix this problem, describe it here.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
GPUs: 8 x RTX 3090
Configs:
click to expand the config
The text was updated successfully, but these errors were encountered: