Monitoring using wandb

[joshlk]

Monitoring using Weights & Biases (wandb). Each worker submits it own report. These are then aggregated using a common group key. The reports are submitted to a eleutherai org account. wandb have kindly provided us with a free org account at my request. Please DM me if you want to be added to it.

Instead of using the normal deepspeed entry point you need to use deepy.py which is in the root dir. This handles generating a common group key and communicating the wandb API key to all workers. Otherwise its a drop in replacement.

For example, here is how it works:

1. Log into wandb wandb login. If you aren't logged in it wont report to wandb.ai but the model will continue as usual
2. Run deepspeed using the deepy.py entry point:

./deepy.py train_pipeline.py --deepspeed --deepspeed_config configs/checkpoints_config.json --model configs/gpt3_small.json	

deepy.py == deepspeed

Example report: https://wandb.ai/eleutherai/neox_train_enwik8/groups/eaWBBEYorwPheZ7s8pWXPb?workspace=user-joshlk

[joshlk]

Not ready to merge quite yet

[joshlk]

Example:

• ./deepy.py train_pipeline.py --deepspeed --deepspeed_config configs/checkpoints_config.json --model configs/gpt3_small.json
• https://wandb.ai/eleutherai/neox_train_pipeline/groups/RRSE66WPNT9aRzmfPDbk6m?workspace=user-joshlk

Branch ready for review and to be merged

[StellaAthena]

Any idea what's going on here?

[joshlk]

Hmmm. Try ‘git pull’

[StellaAthena]

Hmmm. Try ‘git pull’

It says I am up to date

[joshlk]

Odd as deepy.py has been committed into the branch

[StellaAthena]

The first time I build the pod on the main branch and checked out this branch. I tried again building with this branch and now it’s working. A little awkward, but not the end of the world.

I am running the tokenization now and once the model starts actually chaining I’ll merge both this PR and #107

[joshlk]

Here are the detailed steps required to test. It's a bit convoluted due to #107 not being merged:

1. By default a run will add the monitoring to the EleutherAI wandb org account. You can configure this by changing the entry in the model config json. For that to work you need to be added to the EleutherAI wandb org account - I have just sent you an invite now.
2. On your local machine checkout the branch synced-deployment and start a deployment using feature/monitoring_wandb branch's code and image (the image has wandb pip installed):

kubernetes/deploy_k8s.sh feature/monitoring_wandb 4 stellabiderman joshlk/gpt-neox:wandb

Here I'm referencing the docker image joshlk/gpt-neox:wandb which I manually built. But once #107 is merged we can change the deployment script so it uses the docker image from the same branch.

3. You will be dropped into the shell of the main node after the deployment script has finished. Here login to wandb: wandb login. Once Better deployment #107 is merged we can change the deployment script to automatically copy your login details from your local machine to the main node so this step wont be necessary.
4. The best test to first run is with the dummy pipeline as you will get quick feedback. So try running:

./deepy.py train_enwik8.py --deepspeed --deepspeed_config configs/checkpoints_config.json

You then should see something like this in the logs/screen output:

10.141.250.251: wandb: Tracking run with wandb version 0.10.15
10.141.250.251: wandb: Syncing run neox-josh-6d4cbb7ccb-x2f72-5
10.141.250.251: wandb: ⭐️ View project at https://wandb.ai/eleutherai/neox_train_enwik8
10.141.250.251: wandb: 🚀 View run at https://wandb.ai/eleutherai/neox_train_enwik8/runs/1o1lyp45
10.141.250.251: wandb: Run data is saved locally in /app/wandb/run-20210201_093554-1o1lyp45
10.141.250.251: wandb: Run `wandb offline` to turn off syncing.

You can then open https://wandb.ai/eleutherai and select the latest run and you should see it. Let me know if that works 😃

[StellaAthena]

I ran ./deepy.py train_enwik8.py --deepspeed --deepspeed_config configs/checkpoints_config.json successfully. It logs on WandB and things generally look good. Two questions:

1. I am using two nodes, but I see four "groups" on WandB, three of which eventually stop advancing. Are these past runs or something?
2. I got the following warning printout when running the code

/build_dir/src/deepspeed/deepspeed/runtime/pipe/module.py:533: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if rank_repr is not '':

I then tried replacing train_enwik8.py with train.py and it didn't work, but train_pipeline.py did work. I'm going to call that a win and merge both PRs, then work on improving usability.

[joshlk]

The deployment script now automatically uploads your wandb login details to the main node. So you only have to login once on your local machine using wandb login and it should all work automatically.

Branch ready to be merged.