Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
5 Open 59 Closed
5 Open 59 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Eval Harness Refactor Help
#1067 by StellaAthena was closed Feb 11, 2024 updated Feb 11, 2024
4
Automate upload to Pypi for each new release
#1165 by StellaAthena was closed Jan 31, 2024 updated Jan 31, 2024
1
1
Genericize Arguments feature request A feature that isn't implemented yet.
#1084 by StellaAthena was closed Jan 1, 2024 updated Jan 1, 2024
@haileyschoelkopf
2
Push v0.4.0 to PyPI
#1161 by StellaAthena was closed Dec 26, 2023 updated Dec 26, 2023
3
Move to ruff for linting
#1164 by StellaAthena was closed Dec 20, 2023 updated Dec 20, 2023
1
Run isort on codebase, add to CI feature request A feature that isn't implemented yet. good first issue Good for newcomers
#1162 by StellaAthena was closed Dec 20, 2023 updated Dec 20, 2023
1
Implement the QuAC evaluation feature request A feature that isn't implemented yet. good first issue Good for newcomers
#18 by StellaAthena was closed Nov 14, 2023 updated Nov 14, 2023
1 of 2 tasks
1
6
LAMBADA good first issue Good for newcomers validation For validation of task implementations.
#449 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
@StellaAthena
Validate OpenBookQA good first issue Good for newcomers validation For validation of task implementations.
#454 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
@Michionlion
1
Validate HellaSwag good first issue Good for newcomers validation For validation of task implementations.
#452 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
@logan-mo
1
Validate SciQ good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#458 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
Validate Winogrande good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#463 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
Validate TruthfulQA good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#457 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
Validate AI2 Reasoning Challenge good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#462 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
Validate BoolQ good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#461 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
Validate Hendrycks Math good first issue Good for newcomers validation For validation of task implementations.
#451 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
@woog97 @Narmeen07
2
Validate PiQA good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#459 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
Validate MathQA good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#460 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
Validate WSC good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#464 by StellaAthena was closed Nov 8, 2023 updated Nov 8, 2023
Implement GPT-3 style contamination study feature request A feature that isn't implemented yet.
#231 by StellaAthena was closed Nov 1, 2023 updated Nov 1, 2023
@Quentin-Anthony
10
Implement the Natural Questions evaluation feature request A feature that isn't implemented yet. good first issue Good for newcomers
#9 by StellaAthena was closed Aug 21, 2023 updated Aug 22, 2023
1 of 2 tasks
1
19
Add Mac Support
#670 by StellaAthena was closed Aug 6, 2023 updated Aug 6, 2023
2
2
Validate TriviaQA good first issue Good for newcomers validation For validation of task implementations.
#456 by StellaAthena was closed Jun 14, 2023 updated Jun 14, 2023
1
7
Validate MNLI good first issue Good for newcomers validation For validation of task implementations.
#450 by StellaAthena was closed May 11, 2023 updated May 11, 2023
@PatrykNeubauer
5
Multilingual StoryCloze feature request A feature that isn't implemented yet. good first issue Good for newcomers
#241 by StellaAthena was closed May 11, 2023 updated May 11, 2023
1
@juletx
4
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.