Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MMLU multiple choice 5-shot #528

Merged
merged 2 commits into from
Mar 27, 2024
Merged

Add MMLU multiple choice 5-shot #528

merged 2 commits into from
Mar 27, 2024

Conversation

yulinggu-cs
Copy link
Contributor

Added MMLU multiple choice (A/B/C/D) 5-shot variant. This follows the prompt formatting in Eleuther AI Harness.
This will show up in wandb with "len_norm" tag but since the predicted continuation is just a single token, it is the same as acc.

List of tasks available to use in training config:

    - label: mmlu_stem_mc_5shot
      type: downstream
    - label: mmlu_humanities_mc_5shot
      type: downstream
    - label: mmlu_social_sciences_mc_5shot
      type: downstream
    - label: mmlu_other_mc_5shot
      type: downstream

This will run eval on the validation partition, if running on the test partition (10X larger) is desired, add _test to the label name e.g., mmlu_stem_mc_5shot_test

Example:

doc_text: The following are multiple choice questions (with answers) about global facts:

Which of the following pairs of statements are both true (as of 2019)?
A. People tend to be optimistic about their own future and the future of their nation or the world.
B. People tend to be optimistic about their own future but pessimistic about the future of their nation or the world.
C. People tend to be pessimistic about their own future but optimistic about the future of their nation or the world.
D. People tend to be pessimistic about their own future and the future of their nation or the world.
Answer: B

As of 2019, about what percentage of Americans agree that the state is run for the benefit of all the people?
A. 31%
B. 46%
C. 61%
D. 76%
Answer: B

As of 2015, since 1990 forests have ____ in Europe and have ____ in Africa and the Americas.
A. increased, increased
B. increased, decreased
C. decreased, increased
D. decreased, decreased
Answer: B

As of 2019, about what percentage of Russians say it is very important to have free media in our country without government/state censorship?
A. 38%
B. 53%
C. 68%
D. 83%
Answer: A

As of 2017, how many of the world’s 1-year-old children today have been vaccinated against some disease? *
A. 80%
B. 60%
C. 40%
D. 20%
Answer: A

As of 2020, about what percentage of the population in Sub-Saharan Africa, is living on less than $1.90?
A. 10%
B. 20%
C. 40%
D. 80%
Answer:
continuations: [' A', ' B', ' C', ' D']

@OyvindTafjord
Copy link
Contributor

This PR was passing all tests before merging in some small recent changes (771c828), not sure what's going on?

Copy link
Member

@epwalsh epwalsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yulinggu-cs, let's ignore the CI error since it's unrelated.

@epwalsh epwalsh merged commit 71f7014 into main Mar 27, 2024
10 of 11 checks passed
@epwalsh epwalsh deleted the add-mmlu-mc-5shot branch March 27, 2024 23:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants