An implementation of model parallel GPT-3-like models on GPUs, based on the DeepSpeed library. Designed to be able to train models in the hundreds of billions of parameters or larger.
$ pip install -r requirements.txtTest locally
$ python train_enwik8.py