-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement the SuperGLUE evaluation #22
Comments
It looks like almost all of the eval code is written, we are just missing RTE. Once that's done, we can move this to DONE. |
@anishthite Right. But that is missing the evaluation code. The rest of the SuperGLUE task has the evaluation code written. |
Sorry, I missed the evaluation part |
Happy to take the SuperGLUE implementation |
Updated with #91
|
Closing for now as free-form generation is a future problem |
Add `axg` and `axb` to SuperGLUE
Add `axg` and `axb` to SuperGLUE
Add `axg` and `axb` to SuperGLUE
I think the harness has free-form generation now, right? |
Yes, we could implement freeform generation now. |
From the GPT-3 paper
The evaluation code should be modeled after the interface in
lm_eval/base.py
and the example of theBoolQ
task inlm_eval/tasks/suerglue.py
The text was updated successfully, but these errors were encountered: