mmlu evaluation fail

[jxiw]

I try to evaluate model in mmlu, however i get

ValueError: No tasks specified, or no tasks found. Please verify the task names.

for example,

lm_eval \
    --model hf \
    --model_args pretrained=microsoft/phi-2,trust_remote_code=True \
    --tasks mmlu \
    --device cuda:0 \
    --batch_size 8 \
    --output_path output/phi-2 \
    --limit 10 \

it only works for a subset of mmlu,

lm_eval \
    --model hf \
    --model_args pretrained=microsoft/phi-2,trust_remote_code=True \
    --tasks mmlu_abstract_algebra \
    --device cuda:0 \
    --batch_size 8 \
    --output_path output/phi-2 \
    --limit 10 \

i think it is a huge issue. could you please to take a look? it does not appear previously.

[haileyschoelkopf]

Hi @jxiw , what was the problem?

[jxiw]

it because i have a fold called 'mmlu' in the current path. it automatically reads config from that folder instead of using the standard mmlu, but that folder is not for a new config.