Refactored PTH DDP env vars creation in SLURM

[vfdev-5]

Fixes #2202

Related to #2048

Description:

• Refactored PTH DDP env vars creation in SLURM

Check list:

• New tests are added (if a new feature is added)
• New doc strings: description and/or example code are in RST format
• Documentation is updated (if required)

[sdesrozis]

If I understand correctly what is done, the idea is to ensure that in such a case, the user didn't use srun as a mistake

srun python -m torch.distributed.launch ...

Therefore, every process should have a rank, local rank and world size greater or equal to what is defined by slurm.

Is it correct ?

[vfdev-5]

The case 2 is to cover use-case like :

srun -N1 -n1 -G8 python -m torch.distributed.launch\
        --nproc_per_node=8 --nnodes=1 --node_rank=0 \
        --master_addr="localhost" --master_port=1234 \
        main.py

[sdesrozis]

case 2)

RANK >= SLURM_PROCID, LOCAL_RANK >= SLURM_LOCALID means that one process is spawn on the node by srun
but
WORLD_SIZE >= SLURM_NTASKS sounds weird. SLURM_NTASKS is the max number of tasks checked by slurm. If WORLD_SIZE is greater to that value, the scheduler should kill the process, because more than allocated ressources for the job are used. I think it could be an issue using gloo.

What do you think ?

[sdesrozis]

Otherwise, it works on my side.

[vfdev-5]

The example above srun -N1 -n1 allocates SLURM_NTASKS=1, but launcher creates ws=8. That's why WORLD_SIZE >= SLURM_NTASKS. Am I missing something ?

See here as well : https://www.hpcworkshops.com/08-ml-on-parallelcluster/03-distributed-data-parallel.html

[sdesrozis]

Yes but the slurm scheduler can kill the job if more ressources than allocated are used. I suppose it depends on how the scheduler is configured. Imagine you schedule a job defining 4 tasks and in fact you use 8, it could be a big issue in production. Anyway, I think that does not really matter. This is a user constraint, we can't handle.

[sdesrozis]

LGTM !