--device cuda:3 not honored when using --model vllm

[LGLG42]

Hi, I've just found the hard way that the --device cuda:X does not do anything when using --model vllm, the only way to force it to use an specific GPU is to pass 'CUDA_VISIBLE_DEVICES=1' to the "lm_eval" command.

[baberabb]

Hi! yeah that's the only way with vllm unfortunately. Would you like to add a logging message here to inform other users about this limitation?

[johnwee1]

Hi! yeah that's the only way with vllm unfortunately. Would you like to add a logging message here to inform other users about this limitation?

+1, I had to skim through the code and it was only because this issue was raised that I could confidently run without accidentally causing errors (because i had other GPUs that were being used as well.)

for anyone else reading this, CUDA_VISIBLE_DEVICES=1,2,3 as a comma separated list is how you would specify multiple GPUs

[haileyschoelkopf]

We could manually set os.environ["CUDA_VISIBLE_DEVICES"] in the code right?

I'm agnostic as to which solution -- setting the env variable ourselves, or just printing a warning log stating that device won't be respected -- open to feedback from users on which they would find more intuitive.