How to deploy multi-nodes training without hostfile?

[SefaZeng]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I cant get the IP for each node before I started the training. The GPUs are allocated when you are starting a task, so I can't set the hostfile before. Can I use the multi-nodes training just like torch did? Just pass the master_addr and master_port?

[Quentin-Anthony]

You can do one of two things:

• Generate the hostfile using information from your job scheduler (e.g. slurm) that can be parsed by your deepspeed multinode-runner of choice (i.e. one of the following: https://github.com/microsoft/DeepSpeed/blob/master/deepspeed/launcher/constants.py#L6-L9). An example of this for Slurm and openMPI would be:

#!/bin/bash
hostfile=/fsx/$(whoami)/hostfiles/hosts_$SLURM_JOBID
rm $hostfile &> /dev/null # for consecutive calls to this script in interactive jobs

# Iterate over the nodes allocated to your job
for i in `scontrol show hostnames $SLURM_NODELIST`
do
    # For each node, add num_gpus (8 here) as hosts
    echo $i slots=8 >>$hostfile
done

• Alternatively, if your job scheduler provides a launcher (e.g. Slurm has srun), you can add that as a DeepSpeed multinode runner. We have already added a multinode runner with slurm/srun at Add SLURM Multinode Runner microsoft/DeepSpeed#2404 which you're encouraged to use.

[macabdul9]

Hi @Quentin-Anthony @StellaAthena ,

Can't find any documentation on how to use "SLURM Multinode Runner". Can you please provide more information about how to use without hostfile?