Can MoRA be supported? #1043
impredicative
started this conversation in
General
Replies: 1 comment 6 replies
-
Hi @impredicative thanks for starting this discussion. Looks like MoRA does well on continued pretraining tasks, which is an area we've recently been expanding our efforts a bit. At the same time, we are also working on adding some other new features at the moment and may not have the bandwidth to take this on until we have a bit more signal here. Looking at the kongds implementation I see ~6 different methods but the paper seems to use only two of them. Would you be interested in contributing these two (you can even just take eq 6 as a starting point, since it appears simplest)? I'd be happy to provide guidance and code review as needed here. |
Beta Was this translation helpful? Give feedback.
6 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Can the entire technique used in https://github.com/kongds/MoRA be supported?
Beta Was this translation helpful? Give feedback.
All reactions