-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement the adversarially-mined Winogrande evaluation #13
Labels
Comments
Note: HuggingFace includes this in its datasets package. |
Format of the evaluations is different, since this task uses the "partial evaluation" method, where they have the network rate the probability of the context with a missing word/phrase filled in. |
Merged
StellaAthena
added
Eval Set
and removed
feature request
A feature that isn't implemented yet.
labels
Oct 23, 2020
Merged
StellaAthena
added
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
labels
Jan 5, 2021
qmdnls
pushed a commit
to qmdnls/lm-evaluation-harness
that referenced
this issue
Aug 17, 2023
Fix tasks for GEM/mlsum
LZY-the-boys
pushed a commit
to LZY-the-boys/lm-evaluation-harness-fast
that referenced
this issue
Sep 12, 2023
Fix tasks for GEM/mlsum
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
From the GPT-3 paper
The evaluation code should be modeled after the interface in
lm_eval/base.py
and the example of theBoolQ
task inlm_eval/tasks/suerglue.py
The text was updated successfully, but these errors were encountered: