Jst/support multi epoch pp qlora finetuning #144
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
LLaMA-7b 4bit QLoRA(lr=3e-4) finetune loss曲线:
![2023-05-12 14-27-44 的屏幕截图](https://private-user-images.githubusercontent.com/74638604/237889083-cce40660-5cc8-47d4-90d1-09cff63796d2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU2Mjc3MzIsIm5iZiI6MTcyNTYyNzQzMiwicGF0aCI6Ii83NDYzODYwNC8yMzc4ODkwODMtY2NlNDA2NjAtNWNjOC00N2Q0LTkwZDEtMDljZmY2Mzc5NmQyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA2VDEyNTcxMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTZkNTFlNjE2NDEwMmE1MzgxODkyN2QwNGVlZjQ4Y2ZmZmQ1MDcyY2E0YmY2Y2I1NWJlZjUyNWIwNGI3MjhmMzEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.SwK5xQeYZ_SiB-_Hl2AFREoEls6ObTNuvQr52mB7cPc)
LLaMA-65b 4bit QLoRA(lr=1e-4) finetune loss曲线(目前仅1 epoch):
![2023-05-15 10-38-15 的屏幕截图](https://private-user-images.githubusercontent.com/74638604/238230042-9be98c35-d237-441d-a7df-3f9062d140f4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU2Mjc3MzIsIm5iZiI6MTcyNTYyNzQzMiwicGF0aCI6Ii83NDYzODYwNC8yMzgyMzAwNDItOWJlOThjMzUtZDIzNy00NDFkLWE3ZGYtM2Y5MDYyZDE0MGY0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MDYlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTA2VDEyNTcxMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWZmOGU3ZTAxOWExOTU5NzZiNWRmNjA0ZWM3ZDIxMWNhYjhlYzU0ZjAwNTBkYzJmNTg2ZWI2YmRjMGE1YzgzYWQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.1u92Ey03JhOmlxEohniAWUbTdlWO04cObHi8hmkP7LE)