You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
MMLU-Pro is an expanded version of MMLU which also includes a greater number of distractor answer choices.
Implementations should strive to match any "canonical" prompting setups introduced in the MMLU-Pro paper; it would be ideal to implement both a CoT generative (generate_until) variant and a loglikelihood-based (as with mmlu task grouping in our repo) variant.
Contributions would be welcome!
The text was updated successfully, but these errors were encountered:
https://arxiv.org/abs/2406.01574
https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro
MMLU-Pro is an expanded version of MMLU which also includes a greater number of distractor answer choices.
Implementations should strive to match any "canonical" prompting setups introduced in the MMLU-Pro paper; it would be ideal to implement both a CoT generative (
generate_until
) variant and a loglikelihood-based (as withmmlu
task grouping in our repo) variant.Contributions would be welcome!
The text was updated successfully, but these errors were encountered: