Created a draft implementation. Example usage:

./deepy.py pretrain_gpt2.py -d configs ds_pretrain_gpt2.yml eleutherai_cluster.yml

• configs is the directory the config files are in
• ds_pretrain_gpt2.yml, eleutherai_cluster.yml are two JSON-ish (ok they are YAML but its basically JSON with comments) files. deepy takes all config files and merges then (so we can separate out the parameters how we want). Currently eleutherai_cluster.yaml has system stuff and ds_pretrain_gpt2.yml is to do with the model

The parameters in the yaml files are automatically separated into the DS runner, Megatron and DS config file parts. They are then converted into the "old" format and provided to the scripts. I wanted to do it this was so that I made as little changes as possible to the megatron codebase - making it easier in the future to merge upstream changes.

Some parameters are also automatically derived, such as the megatron "fp16" param from the DS runner "fp16" param.

./deepy.py pretrain_gpt2.py -d configs ds_pretrain_gpt2.yml eleutherai_cluster.yml

The example above exactly replicates the parameters used in examples/ds_pretrain_gpt2.sh - try it for yourself. It is not intended to show all possible configurations. I can create such a config later. @ShivanshuPurohit I am going to undo your commit as pipe-parallel-size isn't used in the original example.

Okay, deduplicated the remaining params.

Zero parameters should be set deepspeed style, like so:

   "zero_optimization": {
    "stage": 0,
    "allgather_partitions": True,
    "allgather_bucket_size": 500000000,
    "overlap_comm": False,
    "reduce_scatter": True,
    "reduce_bucket_size": 500000000,
    "contiguous_gradients": False,
    "cpu_offload": False
},

same with optimizer params, like so:

     "type": "Adam",
     "params": {
       "lr": 0.00015,
       "max_grad_norm": 1.0,
       "betas": [0.9, 0.95]
     }
   }

(options are "adam", "onebitadam", "cpu_adam", "cpu_torch_adam")

gradient clipping should be set with "gradient_clipping" instead of clip-grads

and i think that's about it.

Josh and I figured out the batch size related problems - so when doing model parallel world_size according to deepspeed is actually dp_parallel_group_size
so if you have model paralellism (either pipeline or model) of 2, and a world size of 16, deepspeed engine sets the world size to 8.
Before we weren't setting all three params (train_batch, g.a.s, train_microbatch_per_gpu) in conjunction, so never ran into the error.

Should be ready to merge now imo - maybe would be good to get solid documentation first though, to avoid confusion

@sdtblck looks a lot better! Do you think there are other params that would be worth bundling together, deepspeed-style? I mostly have the checkpointing args in mind here, I think.

@StellaAthena i think whether we do checkpointing args deepspeed style or not is inconsequential, really. But i can set it up that way if you think it'd be more user friendly.