Commits

Commits on Dec 30, 2020

added clip grads
mohammad
committedDec 30, 2020
done with refactoring and checks
mohammad
committedDec 30, 2020
add multi-tensor-apply to clip grad
mohammad
committedDec 30, 2020
moved grad scaling to multi-tensor-apply
mohammad
committedDec 30, 2020
further refactor, matching old results
mohammad
committedDec 30, 2020
slight refactor of clip grads
mohammad
committedDec 30, 2020

Commits on Dec 28, 2020

found a bug in l2 norm calculation
Mohammad
committedDec 28, 2020

Commits on Dec 27, 2020

moved module to model and removed fp16
mohammad
committedDec 27, 2020
load and save state dicts added
mohammad
committedDec 27, 2020

Commits on Dec 25, 2020

further refactoring
mohammad
committedDec 25, 2020
grads is removed from mpu
mohammad
committedDec 25, 2020
clip grad fixed and moved to optimizer
mohammad
committedDec 25, 2020
moved entire optimizer build and tested
mohammad
committedDec 25, 2020
megatron optimizer tested, before working on clip grad
mohammad
committedDec 25, 2020
fp32 is also working
mohammad
committedDec 25, 2020
working on the new optimizer
mohammad
committedDec 25, 2020

Commits on Dec 23, 2020

Merge branch 'validation_loss_after_checkpoint' into 'main'
Mohammad Shoeybi
committedDec 23, 2020
Checkpoint should be saved only after evaluation pass is run to make sure validation losses are identical after loading checkpoint
deepakn94
committedDec 23, 2020

Commits on Dec 22, 2020

Merge branch 'fp32_residual_conn' into 'main'
deepakn94
committedDec 22, 2020
Address Deepak's comments

mshoeybi
authored and
deepakn94
committedDec 22, 2020
Add residual connection in fp32 machinery

mohammad
authored and
deepakn94
committedDec 22, 2020
Add the option for fp32 residual connection (fp32 residual connection machinery still needs to be added)

mohammad
authored and
deepakn94
committedDec 22, 2020

Commits on Dec 21, 2020

Merge branch 'pipeline_parallel_main' into 'main'
deepakn94
committedDec 21, 2020

Commits on Dec 19, 2020

Quick fix for pipeline tasks to get learning rate correct

jaredcasper
authored and
deepakn94
committedDec 19, 2020
Added a comment to justify 80 percent

mohammad
authored and
deepakn94
committedDec 19, 2020
Move args writer to the beginning of training

mohammad
authored and
deepakn94
committedDec 19, 2020
Cleaned up load/save checkpoint printing

mohammad
authored and
deepakn94
committedDec 19, 2020
Fix TensorBoard writes

mohammad
authored and
deepakn94
committedDec 19, 2020
Fix loss addition in TensorBoard

mshoeybi
authored and
deepakn94
committedDec 19, 2020
Some bugfixes

mshoeybi
authored and
deepakn94
committedDec 19, 2020
Last epoch should not be globally shuffled

mshoeybi
authored and
deepakn94
committedDec 19, 2020
Address Jared's comments

mshoeybi
authored and
deepakn94
committedDec 19, 2020
Fix some bugs, add exit-duration capability

mshoeybi
authored and
deepakn94
committedDec 19, 2020
Add comment describing _PIPELINE_GLOBAL_RANKS

jaredcasper
authored and
deepakn94
committedDec 19, 2020
Fix text generation without recompute

jaredcasper
authored and
deepakn94
committedDec 19, 2020