Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MoE Support #677

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

MoE Support #677

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
3327364
Latest DeepSpeed initial patches
Sep 2, 2022
4f14166
Pass args fix and update reqs
Sep 2, 2022
f65c3ee
Initial MoE work
Quentin-Anthony Sep 19, 2022
2899b43
Delete requirements.txt
Quentin-Anthony Sep 19, 2022
919899e
Formatting
Quentin-Anthony Sep 19, 2022
54c9e92
Revert "Delete requirements.txt"
Quentin-Anthony Sep 19, 2022
276221c
More formatting
Quentin-Anthony Sep 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Latest DeepSpeed initial patches
  • Loading branch information
anthony.301 committed Sep 2, 2022
commit 33273640261fcdd90adacfa0a6491b5fabade088
4 changes: 1 addition & 3 deletions megatron/initialize.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,6 @@
import deepspeed
import inspect

from deepspeed.utils import distributed


def initialize_megatron(neox_args, allow_no_cuda=False):
"""Set initialize distributed and set autoresume and random seeds.
Expand Down Expand Up @@ -148,7 +146,7 @@ def _initialize_distributed(neox_args):
neox_args.local_rank = device
torch.cuda.set_device(device)

distributed.init_distributed(
deepspeed.init_distributed(
dist_backend=neox_args.distributed_backend,
auto_mpi_discovery=True,
distributed_port=os.getenv("MASTER_PORT", "6000"),
Expand Down
3 changes: 1 addition & 2 deletions requirements/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,10 @@
git+https://github.com/EleutherAI/DeeperSpeed.git@eb7f5cff36678625d23db8a8fe78b4a93e5d2c75#egg=deepspeed
einops==0.3.0
ftfy==6.0.1
lm_dataformat==0.0.20
lm_eval==0.2.0
mpi4py==3.0.3
numpy==1.22.0
pybind11==2.6.2
mpi4py==3.0.3
regex
sentencepiece
six
Expand Down