Error Encountered in Multi-Node Evaluation Using Distributed Arguments

[jdy18]

I encountered an issue while attempting to perform a multi-node evaluation using PyTorch's torchrun with specific distributed arguments. Below is the command I used, including the distributed arguments setup and the execution command:

=

DISTRIBUTED_ARGS=" \
    --nproc_per_node 3 \
    --nnodes 4 \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT"

torchrun $DISTRIBUTED_ARGS run.py \
    --data MME MMBench_DEV_EN MMBench_DEV_CN CCBench SEEDBench_IMG MMMU_DEV_VAL MathVista_MINI HallusionBench LLaVABench \
    MMBench_TEST_EN MMBench_TEST_CN \
    --model llava

Upon execution, I received the following error message:

RUN - ERROR - No such file or directory: './llava/312_MME.pkl'
It seems like only the .pkl file on node0 was saved correctly. Only 012,112,212 was saved.
Thank you in advance for your assistance!

[kennymckormick]

Hi, @jdy18 ,
Currently, we have not supported multi-node test yet. I would further check if it's easy to add this new feature.

[jdy18]

Hi, @jdy18 , Currently, we have not supported multi-node test yet. I would further check if it's easy to add this new feature.

I have successfully resolved the issue. The function get_rank_and_world_size() should return rank instead of local_rank, and the call to torch.cuda.set_device(rank) should be modified to torch.cuda.set_device(local_rank).

[kennymckormick]

Great, are you willing to create a PR to support this new feature? I think it's compatible with the original single node multi-GPU parallel inference.

[kennymckormick]

The modification has been updated into the main branch.