Add MoE #1129

yang · 2024-01-23T22:20:44Z

Closes #479

Quentin-Anthony

Tested and working. Thanks @yang !

megatron/training.py

configs/125M-moe.yml

Thanks to dayofthepenguin for extensive testing Closes EleutherAI#479

yang requested a review from Quentin-Anthony as a code owner January 23, 2024 22:20

yang force-pushed the moe2 branch 7 times, most recently from 03c2cb8 to 4362683 Compare January 30, 2024 18:32

yang force-pushed the moe2 branch from 4362683 to 4c8849d Compare February 25, 2024 07:47

Quentin-Anthony previously approved these changes Mar 4, 2024

View reviewed changes

Quentin-Anthony dismissed their stale review via 99eeaaf March 4, 2024 16:03

Quentin-Anthony reviewed Mar 4, 2024

View reviewed changes

megatron/training.py Outdated Show resolved Hide resolved

yang force-pushed the moe2 branch 2 times, most recently from 6f95a29 to bab8e4f Compare March 4, 2024 23:47

StellaAthena reviewed Mar 6, 2024

View reviewed changes

configs/125M-moe.yml Show resolved Hide resolved

yang force-pushed the moe2 branch 2 times, most recently from a14cedd to 89e16e8 Compare March 6, 2024 21:30

Add DeepSpeed MoE

4825f1a

Thanks to dayofthepenguin for extensive testing Closes EleutherAI#479

yang force-pushed the moe2 branch from 6c8751d to 4825f1a Compare March 6, 2024 21:38

Update NeoXArgs docs automatically

7ee6c52

yang requested a review from Quentin-Anthony March 6, 2024 21:40

Quentin-Anthony and others added 2 commits March 6, 2024 21:11

pre-commit

ae06be5

Update NeoXArgs docs automatically

bf6dd4f

Quentin-Anthony approved these changes Mar 7, 2024

View reviewed changes

Quentin-Anthony merged commit 86758c3 into EleutherAI:main Mar 7, 2024
2 checks passed

tf-nv mentioned this pull request Mar 7, 2024

MoE loss variable not defined in gpt j residual code path #1174

Closed

haileyschoelkopf mentioned this pull request May 16, 2024

Conversion script bugfixes #1218

Merged

Provide feedback