Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Pretrained language model with 100B parameters

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.