Including AdaBound in the list of Optimizers.

[AD2605]

🚀 Feature

Hello all,
I would like to add Adabound to the list of existing optimizers in the torch.optim module.
Here is the link to the paper - https://openreview.net/pdf?id=Bkg3g2R9FX

Motivation

In most of my models, I realized Adam started off good but never converged or got stuck even after scheduling the learning rate, and if I started with SGD, the loss turned into NaNs soon. Out of curiosity, I changed the optimizer from Adam to SGD in the middle of the training and noticed better results, and upon researching more I came across the paper mentioned above. Therefore I believe it would be good to have the optimizer listed in PyTorch by default

Pitch

I have already implemented the optimizer, once I get the green flag from your side, I will send the pull request.

cc @vincentqb

[albanD]

Hi, sorry for the delay.
We will be happy to add this new optimizer yes.

We'll review the PR asap.

[AD2605]

Sure !

Thanks a lot !

[hyperkai]

Hey, when will AdaBound() be officially added to the list of optimizers? I cannot find it in the doc.

[AD2605]

Hi @hyperkai

This PR went because of me not being able to being on top of it.
Thanks for the ping.

I can put it up shortly again