Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lion Optimizer #1012

Closed
wants to merge 2 commits into from
Closed

Lion Optimizer #1012

wants to merge 2 commits into from

Conversation

adi-kmt
Copy link
Contributor

@adi-kmt adi-kmt commented Aug 13, 2023 •
edited
Loading

In relation to this issue

TODO:

  • Do you require the trition portion to be ported as well?

@adi-kmt adi-kmt requested a review from a team as a code owner August 13, 2023 06:25
@adi-kmt adi-kmt marked this pull request as draft August 13, 2023 06:25
@CLAassistant
Copy link

CLAassistant commented Aug 13, 2023 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@adi-kmt adi-kmt marked this pull request as ready for review August 13, 2023 07:23
@adi-kmt adi-kmt changed the title [WIP] Lion Optimizer Lion Optimizer Aug 21, 2023
@Quentin-Anthony
Copy link
Member

Thanks for working on this! Have you:

  1. Tested the Lion optimizer with a very small model and seen loss decrease? Have you compared this decrease to Adam and seen the expected memory benefits? We tried to integrate Adafactor here but didn't see the expected benefits and therefore didn't merge.
  2. Sharded the optimizer with DeepSpeed ZeRO (just stage 1 is fine) and verified loss still decreases and nothing breaks

@StellaAthena
Copy link
Member

@kamathis4 Hey, I wanted to follow up on this.

@adi-kmt
Copy link
Contributor Author

adi-kmt commented Oct 19, 2023

Sorry for the delay @StellaAthena, please assign it to someone else. Been really busy.

@StellaAthena StellaAthena linked an issue Oct 19, 2023 that may be closed by this pull request
Incorporation of LION Optimizer in GPT-NeoX #950
Closed
@StellaAthena StellaAthena added good first issue Good for newcomers help wanted This issue needs assistance labels Oct 19, 2023
@ocss884
Copy link

ocss884 commented Oct 19, 2023

@StellaAthena Hi Stella, I can help with the testing. I have sent you a DM in Discord.

@andylolu2 andylolu2 mentioned this pull request Oct 20, 2023
Merged
@andylolu2
Copy link
Contributor

Hey, I picked up with PR: fixed some bugs and did some tiny experiments #1062. Should we close this PR?
(My discord handle is BjergerK1ng#2545)

@Quentin-Anthony
Copy link
Member

Resolved by #1062

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Quentin-Anthony Quentin-Anthony Awaiting requested review from Quentin-Anthony Quentin-Anthony was automatically assigned from EleutherAI/pm-gptneo

@ShivanshuPurohit ShivanshuPurohit Awaiting requested review from ShivanshuPurohit ShivanshuPurohit was automatically assigned from EleutherAI/pm-gptneo

Assignees
No one assigned
Labels
good first issue Good for newcomers help wanted This issue needs assistance
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Incorporation of LION Optimizer in GPT-NeoX
6 participants
@adi-kmt @CLAassistant @Quentin-Anthony @StellaAthena @ocss884 @andylolu2