Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the HellaSwag evaluation #7

Closed
2 tasks done
StellaAthena opened this issue Sep 16, 2020 · 2 comments · Fixed by #43
Closed
2 tasks done

Implement the HellaSwag evaluation #7

StellaAthena opened this issue Sep 16, 2020 · 2 comments · Fixed by #43
Assignees
Labels
feature request A feature that isn't implemented yet. good first issue Good for newcomers

Comments

@StellaAthena
Copy link
Member

StellaAthena commented Sep 16, 2020

From the GPT-3 paper

The HellaSwag dataset [ZHB+19] involves picking the best ending to a story or set of instructions. The examples were adversarially mined to be difficult for language models while remaining easy for humans (who achieve 95.6% accuracy). GPT-3 achieves 78.1% accuracy in the one-shot setting and 79.3% accuracy in the few-shot setting, outperforming the 75.4% accuracy of a fine-tuned 1.5B parameter language model [ZHR+19] but still a fair amount lower than the overall SOTA of 85.6% achieved by the fine-tuned multi-task model ALUM.

  • Data processing code implemented
  • Evaluation implemented

The evaluation code should be modeled after the interface in lm_eval/base.py and the example of the BoolQ task in lm_eval/tasks/suerglue.py

@StellaAthena StellaAthena added the feature request A feature that isn't implemented yet. label Sep 16, 2020
@StellaAthena StellaAthena added this to To do in Implementing Evaluations via automation Sep 16, 2020
@StellaAthena StellaAthena changed the title Implement the HellaSwag test Implement the HellaSwag evaluation Sep 16, 2020
@cfoster0
Copy link
Contributor

cfoster0 commented Oct 1, 2020

Note: HuggingFace includes this in its datasets package.

https://huggingface.co/datasets/hellaswag

@cfoster0
Copy link
Contributor

cfoster0 commented Oct 5, 2020

Working on adding this for dedupe.

@StellaAthena StellaAthena moved this from To do to In progress in Implementing Evaluations Oct 5, 2020
@StellaAthena StellaAthena linked a pull request Oct 23, 2020 that will close this issue
@StellaAthena StellaAthena added Eval Set and removed feature request A feature that isn't implemented yet. labels Oct 23, 2020
@StellaAthena StellaAthena moved this from In progress to Data integrated, Eval not done in Implementing Evaluations Oct 23, 2020
@StellaAthena StellaAthena reopened this Jan 5, 2021
@StellaAthena StellaAthena added feature request A feature that isn't implemented yet. good first issue Good for newcomers labels Jan 5, 2021
@leogao2 leogao2 moved this from In Progress to To do in Implementing Evaluations Jan 28, 2021
@anishthite anishthite self-assigned this Feb 7, 2021
@leogao2 leogao2 moved this from To do, Evaluations to Implement to Done, evaluations in Implementing Evaluations Feb 8, 2021
@leogao2 leogao2 assigned jon-tow and unassigned anishthite Feb 8, 2021
@leogao2 leogao2 closed this as completed Feb 8, 2021
leogao2 pushed a commit that referenced this issue Mar 28, 2021
Fork update and long-overdue SQuAD fixes
StellaAthena pushed a commit that referenced this issue Apr 29, 2022
qmdnls pushed a commit to qmdnls/lm-evaluation-harness that referenced this issue Aug 17, 2023
Fork update and long-overdue SQuAD fixes
qmdnls pushed a commit to qmdnls/lm-evaluation-harness that referenced this issue Aug 17, 2023
LZY-the-boys pushed a commit to LZY-the-boys/lm-evaluation-harness-fast that referenced this issue Sep 12, 2023
Fork update and long-overdue SQuAD fixes
LZY-the-boys pushed a commit to LZY-the-boys/lm-evaluation-harness-fast that referenced this issue Sep 12, 2023
lintangsutawika pushed a commit that referenced this issue Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request A feature that isn't implemented yet. good first issue Good for newcomers
Projects
No open projects
Implementing Evaluations
  
Done, evaluations
Development

Successfully merging a pull request may close this issue.

5 participants