[Feature] Implement & Optimize a few optimizer options

[kozistr]

Problem (Why?)

implement & optimize a few features

Solution (What/How?)

• Move step parameter from state to group. (to reduce computation cost & memory)
• load betas by group, not parameter.
• refactor to in-place operations.
• Adan optimizer.
  • Support max_grad_norm.
• Lamb optimizer.
  • Support gradient averaging.
• Lars optimizer
  • Support dampening, nesterov parameters.
  • Fix logic when momentum is 0.

Other changes (bug fixes, small refactors)

nope

Notes

bump version to v2.2.1

[codecov-commenter]

Codecov Report

Merging #95 (27d6b99) into main (f6baa63) will increase coverage by 0.02%.
The diff coverage is 100.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@            Coverage Diff             @@
##             main      #95      +/-   ##
==========================================
+ Coverage   97.73%   97.76%   +0.02%     
==========================================
  Files          37       37              
  Lines        2520     2551      +31     
==========================================
+ Hits         2463     2494      +31     
  Misses         57       57              

Impacted Files	Coverage Δ
pytorch_optimizer/optimizer/gc.py	100.00% <ø> (ø)
pytorch_optimizer/optimizer/sgdp.py	100.00% <ø> (ø)
pytorch_optimizer/base/optimizer.py	97.56% <100.00%> (+0.12%)	⬆️
pytorch_optimizer/optimizer/adabelief.py	97.29% <100.00%> (ø)
pytorch_optimizer/optimizer/adabound.py	100.00% <100.00%> (ø)
pytorch_optimizer/optimizer/adai.py	100.00% <100.00%> (ø)
pytorch_optimizer/optimizer/adamp.py	100.00% <100.00%> (ø)
pytorch_optimizer/optimizer/adan.py	100.00% <100.00%> (ø)
pytorch_optimizer/optimizer/adapnm.py	100.00% <100.00%> (ø)
pytorch_optimizer/optimizer/diffgrad.py	100.00% <100.00%> (ø)
... and 10 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.