Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Adafactor #209

Closed
sdtblck opened this issue Apr 5, 2021 · 1 comment
Closed

Add Adafactor #209

sdtblck opened this issue Apr 5, 2021 · 1 comment
Assignees
@Quentin-Anthony
Labels
feature request New feature or request
Projects
1T or BUST

Comments

@sdtblck
Copy link
Contributor

sdtblck commented Apr 5, 2021

should be a pretty easy addition
@sdtblck sdtblck added the feature request New feature or request label Apr 5, 2021
@cppntn cppntn linked a pull request Apr 15, 2021 that will close this issue
Add Adafactor #236
Closed
@StellaAthena StellaAthena added this to To do in 1T or BUST via automation May 14, 2021
@Quentin-Anthony Quentin-Anthony self-assigned this Oct 26, 2021
@StellaAthena StellaAthena linked a pull request Nov 3, 2021 that will close this issue
[WIP] Add Adafactor Optimizer #453
Closed
@StellaAthena
Copy link
Member

StellaAthena commented Dec 4, 2021 •
edited
Loading

As discussed in the linked PR, there appears to be something wrong with the Adafactor implementation as it runs slower than Ada. If you are interested in this feature I recommend checking out the discussion on that PR and seeing if you can resolve the problem.

1T or BUST automation moved this from To do to Done Mar 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Quentin-Anthony Quentin-Anthony

Labels
feature request New feature or request
Projects
1T or BUST
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add Adafactor EleutherAI/gpt-neox
[WIP] Add Adafactor Optimizer EleutherAI/gpt-neox
4 participants
@Quentin-Anthony @StellaAthena @cppntn @sdtblck