-
Notifications
You must be signed in to change notification settings - Fork 53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
finetune error about model size #24
Comments
Thank you for your interest in our work. I notice another person also faced the same error, however not yet sure about what causing it. Could you please try using Thank You |
Hi @mmaaz60 |
May I know which deepspeed version are you using? Try to use version |
Hi @mmaaz60 it works when moving the mm_projector initialization code outside for loop. Thank you for your patience in solving the issue. |
Hi, I encountered the same issue while fine-tuning LoRA. Could you please share your solution if possible? |
Issue Title: use the finetune script but meet error
Environment
Issue Description
when i used the llava-pp codebase and the finetune script to finetune our model, it appeared that :
but when i use the llava official codebase and script , i didn't meet that error, which is weird! it seems that the deepspeed zero3 has some bug and can't gather the sharded parameters.
I've tried various approaches in the deepspeed issues and transformers issues to fix this issue but haven't been successful. Any help would be greatly appreciated!
Full Script
Full StackTrace
The text was updated successfully, but these errors were encountered: