[lora] support lora for Gemini #5001
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fridge003
force-pushed
the
feature/lora
branch
from
ac56dee
to
e187290
Compare
Fridge003
force-pushed
the
feature/lora
branch
from
5e46e2f
to
50ebc1a
Compare
Fridge003
force-pushed
the
feature/lora
branch
6 times, most recently
from
69136b6
to
3bf7e1c
Compare
Fridge003
force-pushed
the
feature/lora
branch
from
3bf7e1c
to
49c5d73
Compare
Fridge003
force-pushed
the
feature/lora
branch
from
fb1a59b
to
12e1ae9
Compare
ver217
reviewed
Nov 9, 2023
| peft_config.base_model_name_or_path = peft_model.base_model.model.__dict__.get("name_or_path", None) | ||
|
|
||
| inference_mode = peft_config.inference_mode | ||
| peft_config.inference_mode = True |
There was a problem hiding this comment.
Why set inference mode to True before saving?
There was a problem hiding this comment.
Peft will set it to True when saving, and I didn't change this behavior.
| @@ -142,7 +142,6 @@ def _flatten_grad_args(args) -> Tuple[list, list, List[bool], TreeSpec]: | |||
| grad_args.append(arg) | |||
| else: | |||
| other_args.append(arg) | |||
| assert len(grad_args) > 0 | |||
There was a problem hiding this comment.
Why remove the assertation? If there are no grad args, post-backward hook may be not triggered.
There was a problem hiding this comment.
They are plenty of parameters with requires_grad being False when lora is enabled. So len(grad_args) often equals to zero during training.
Fridge003
force-pushed
the
feature/lora
branch
from
5193428
to
6cece1d
Compare
|
any update of this work? |
|
Looking forward to the feature implemented in this PR！ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
📌 Checklist before creating the PR
[doc/gemini/tensor/...]: A concise description🚨 Issue number
resolved #4929
📝 What does this PR do?
💥 Checklist before requesting a review
⭐️ Do you enjoy contributing to Colossal-AI?
Tell us more if you don't enjoy contributing to Colossal-AI.