📑 Paper | 🤗 Leaderboard & 🤗 Dataset

HF_MODEL_ID="Magpie-Align/Llama-3-8B-Magpie-Align-v0.1" # example model id 
MODEL_PRETTY_NAME="Llama-3-8B-Magpie-Align-v0.1" # example model name
NUM_GPUS=4 # depending on your hardwares;
# do inference on WildBench 
bash scripts/_common_vllm.sh $HF_MODEL_ID $MODEL_PRETTY_NAME $NUM_GPUS 
# submit to OpenAI for eval (WB-Score)
bash evaluation/run_score_eval_batch.sh ${MODEL_PRETTY_NAME} 
# check the batch job status
python src/openai_batch_eval/check_batch_status_with_model_name.py ${MODEL_PRETTY_NAME} 
# show the table 
bash leaderboard/show_eval.sh score_only 

conda create -n zeroeval python=3.10
conda activate zeroeval
# pip install vllm -U # pip install -e vllm 
pip install vllm==0.5.1
pip install -r requirements.txt

bash scripts/_common_vllm.sh [hf_model_id] [model_pretty_name] [num_gpus]
# bash scripts/_common_vllm.sh m-a-p/neo_7b_instruct_v0.1 neo_7b_instruct_v0.1 4 # example
# 1st arg is hf_name; 2nd is the pretty name; 3rd is the number of shards (gpus)

bash evaluation/run_score_eval_batch.sh ${MODEL_PRETTY_NAME}

bash leaderboard/show_eval.sh score_only

We suggest to use OpenAI's Batch Mode for evaluation, which is faster, cheaper and more reliable.

You can:

• 1. Run bash evaluation/run_all_eval_batch.sh ${MODEL_PRETTY_NAME}to submit the eval jobs.; Or if you only want to do scoring, running bash evaluation/run_score_eval_batch.sh to submit the eval jobs for only doing the WB Score. (about $5 per model)
• 2. Run python src/openai_batch_eval/check_batch_status_with_model_name.py ${MODEL_PRETTY_NAME} to track the status of the batch jobs.
• 3. Step 2 will download the results when batch jobs are finished, and then you can view the results (see next section).

Remarks

• ${MODEL_PRETTY_NAME} should be the same as the script name without the .sh extension.
• You can also track the progress of your batch jobs here: https://platform.openai.com/batches. The maximum turnaround time is 24 hours, but it is usually much faster depending on the queue and rate limits.
• If you'd like to have more control on the evaluation methods, the detail steps are illustrated in EVAL.md.

When Step 3 in the above section is finished, you can view the results by running the following commands:

bash leaderboard/show_eval.sh # run all and show the main leaderboard
python leaderboard/show_table.py --mode main  # (optional) to show the main leaderboard w/o recomputing 
python leaderboard/show_table.py --mode taskwise_score # (optional) to show the taskwise score

To analyze the correlation between WildBench (v2) and human evaluation, we consider the correlation between different metrics and human-based Chatbot Arena Elo scores (until 2024-05-20 on Hard-English split). We find that the WB Reward-Mix has the highest correlation. Please find the pearson correlation coefficients below:

• Top Models: ['gpt-4-turbo-2024-04-09', 'claude-3-opus-20240229', 'Meta-Llama-3-70B-Instruct', 'claude-3-sonnet-20240229', 'mistral-large-2402', 'Meta-Llama-3-8B-Instruct']
• All Models: ['gpt-4-turbo-2024-04-09', 'claude-3-opus-20240229', 'Meta-Llama-3-70B-Instruct', 'Qwen1.5-72B-Chat', 'claude-3-sonnet-20240229', 'mistral-large-2402', 'dbrx-instruct@together', 'Mixtral-8x7B-Instruct-v0.1', 'Meta-Llama-3-8B-Instruct', 'tulu-2-dpo-70b', 'Llama-2-70b-chat-hf', 'Llama-2-7b-chat-hf', 'gemma-7b-it', 'gemma-2b-it']

• openchat/openchat-3.6-8b-20240522
• gemma-2
• SimPO-v0.2
• Qwen2-7B-Chat
• LLM360/K2-Chat
• DeepSeek-V2-Code
• Yi-large-preview
• THUDM/glm-4-9b-chat
• chujiezheng/neo_7b_instruct_v0.1-ExPO
• ZhangShenao/SELM-Llama-3-8B-Instruct-iter-3
• m-a-p/neo_7b_instruct_v0.1
• Reka Flash
• DeepSeekV2-Chat
• Reka Core
• Yi-Large (via OpenAI-like APIs)
• chujiezheng/Llama-3-Instruct-8B-SimPO-ExPO
• chujiezheng/Starling-LM-7B-beta-ExPO
• Gemini 1.5 series
• Qwen2-72B-Instruct
• ZhangShenao/SELM-Zephyr-7B-iter-3
• NousResearch/Hermes-2-Theta-Llama-3-8B
• princeton-nlp/Llama-3-Instruct-8B-SimPO
• Command-R-plus
• Phi-3 series

Create an Issue if you'd like to add a model that you wanna see on our leaderboard!

• support models via openai-style apis

• Show task categorized results

@misc{lin2024wildbench,
    title={WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild},
    author={Bill Yuchen Lin and Yuntian Deng and Khyathi Chandu and Faeze Brahman and Abhilasha Ravichander and Valentina Pyatkin and Nouha Dziri and Ronan Le Bras and Yejin Choi},
    year={2024},
    eprint={2406.04770},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2406.04770}
}