-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to run the finetuned model with LoRA adapters. #59
Comments
Hi @thisurawz1, I was wondering if you were available for a call or text, we are currently experiencing some issues when fine tuning with finetune_lora.sh file, and was wondering if we could use your guidance. I have a discord as well if you prefer, let me know what works best for you |
You can contact me on Discord - "wick6309". However, I'm not very active on Discord and mainly use WeChat. Anyway, I've posted below all the issues I encountered and their solutions for everyone's reference. I mainly used the QLoRA script and did a fine-tuning as a trial run. My dataset was quite small, around 229 samples (image and text). I encountered the following issues while doing the fine-tuning. I used 1 A100 40GB GPU, but the VRAM was not enough to run the QLoRA script with a batch size of 4, so I had to change it to 2. 1 Adjust the number of GPUs available in your PC
2 Hugging Face offline mode error
3 Cannot access "mistralai/Mistral-7B-Instruct-v0.2" as it is a private repo
4 mm_projector.bin couldn't be found
5 change the dataset path and folder.
6 NCCL error/ CUDA error/ Not enough VRAM
|
Hey @thisurawz1 thanks a lot for the reply, it made things clear, I am working with @lucasxu777 on this so if you could add him that would be great since he has wechat! I have also added you on discord as well ".yogiii" is my username. |
It was solved here: #32 |
noted. ill add you |
Thanks. ill add your friend. is there any proper guide on how to do the inference with the lora fine tuned model. |
i have successfully fine-tuned the model using QLORA for a custom use case. now i have the LoRA adapters and can you tell how to use it for the inference. maybe merge lora weights with the original model and do the inference.
The text was updated successfully, but these errors were encountered: