Can MoRA be supported?

[impredicative]

Can the entire technique used in https://github.com/kongds/MoRA be supported?

[ebsmothers]

Hi @impredicative thanks for starting this discussion. Looks like MoRA does well on continued pretraining tasks, which is an area we've recently been expanding our efforts a bit. At the same time, we are also working on adding some other new features at the moment and may not have the bandwidth to take this on until we have a bit more signal here. Looking at the kongds implementation I see ~6 different methods but the paper seems to use only two of them. Would you be interested in contributing these two (you can even just take eq 6 as a starting point, since it appears simplest)? I'd be happy to provide guidance and code review as needed here.

[ebsmothers]

Hi @leighgable thanks for offering to take this on. We would be happy to have your contribution here. You can probably get started by adding an nn.Module MoRALayer with corresponding unit test in torchtune/modules/peft/mora.py. You can follow the design of LoRALinear for inspiration here. Let me know if that sounds reasonable to you.

[leighgable]

@ebsmothers I've put together an initial MoRALinear layer, but I'm having trouble running the tests due to import errors. Could I bother you for some feedback or hints? I could make a PR, or just send you a link to the fork I made...

[RdoubleA]

Hi @leighgable, feel free to share the import error you're running into here and a link to your fork. Once you're able to get it tested properly you can open a PR. @ebsmothers is off this week, so I'm happy to answer any questions and help review your PR. You can also message on the discord channel and tag me (@RdoubleA)

[pbontrager]

@leighgable if this is your first time contributing, I'd recommend looking through the contributors guide and make sure you installed the dev dependencies.

[leighgable]

Hello @RdoubleA ! My problem was indeed a missing dev dependency. Now all the tests are passing except one datatype mis-match for the test_qmora_parity function. If you have a moment to look at it, I have not been able to resolve it despite adding dot-to-dtype everywhere. My fork is here .