The output of ceval is not as the same format at the official version?

[ChuanhongLi]

As https://github.com/hkust-nlp/ceval/blob/main/README_zh.md shows:

but the result of ours is:

The script is:

TASKS="ceval-valid*"
BATCH_SIZE=1
NUM_SHOTS=5

MODEL="" # model path
API=vllm
CUDA_VISIBLE_DEVICES=7 lm_eval \
    --model ${API} \
    --model_args pretrained=${MODEL},dtype="half", gpu_memory_utilization=0.9 \
    --tasks ${TASKS} \
    --batch_size ${BATCH_SIZE} \
    --num_fewshot ${NUM_SHOTS} \
    --trust_remote_code \
    --output_path ./result

How can I get the same ouput as the official version?

Thanks!

[LSinev]

Since GPT-4 and ChatGPT API replies may change every day, week or month, there may be no way to get the same output as reported 8 months ago (according to the last change date of the README_zh.md link you provided).

You can also try to find exact version of lm-eval code they used (probably some 0.3.0 like with updates) and check reproducibility.