Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
5 Open 59 Closed
5 Open 59 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Automate upload to Pypi for each new release
#1165 by StellaAthena was closed Jan 31, 2024
1
1
Move to ruff for linting
#1164 by StellaAthena was closed Dec 20, 2023
1
Run isort on codebase, add to CI feature request A feature that isn't implemented yet. good first issue Good for newcomers
#1162 by StellaAthena was closed Dec 20, 2023
1
Push v0.4.0 to PyPI
#1161 by StellaAthena was closed Dec 26, 2023
3
Genericize Arguments feature request A feature that isn't implemented yet.
#1084 by StellaAthena was closed Jan 1, 2024
@haileyschoelkopf
2
Eval Harness Refactor Help
#1067 by StellaAthena was closed Feb 11, 2024
4
Add Mac Support
#670 by StellaAthena was closed Aug 6, 2023
2
2
Validate WSC good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#464 by StellaAthena was closed Nov 8, 2023
Validate Winogrande good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#463 by StellaAthena was closed Nov 8, 2023
Validate AI2 Reasoning Challenge good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#462 by StellaAthena was closed Nov 8, 2023
Validate BoolQ good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#461 by StellaAthena was closed Nov 8, 2023
Validate MathQA good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#460 by StellaAthena was closed Nov 8, 2023
Validate PiQA good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#459 by StellaAthena was closed Nov 8, 2023
Validate SciQ good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#458 by StellaAthena was closed Nov 8, 2023
Validate TruthfulQA good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#457 by StellaAthena was closed Nov 8, 2023
Validate TriviaQA good first issue Good for newcomers validation For validation of task implementations.
#456 by StellaAthena was closed Jun 14, 2023
1
7
HellaSwag duplicate This issue or pull request already exists. good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#455 by StellaAthena was closed May 2, 2023
1
Validate OpenBookQA good first issue Good for newcomers validation For validation of task implementations.
#454 by StellaAthena was closed Nov 8, 2023
@Michionlion
1
MMMLU good first issue Good for newcomers help wanted Contributors and extra help welcome. validation For validation of task implementations.
#453 by StellaAthena was closed May 8, 2023
Validate HellaSwag good first issue Good for newcomers validation For validation of task implementations.
#452 by StellaAthena was closed Nov 8, 2023
@logan-mo
1
Validate Hendrycks Math good first issue Good for newcomers validation For validation of task implementations.
#451 by StellaAthena was closed Nov 8, 2023
@woog97 @Narmeen07
2
Validate MNLI good first issue Good for newcomers validation For validation of task implementations.
#450 by StellaAthena was closed May 11, 2023
@PatrykNeubauer
5
LAMBADA good first issue Good for newcomers validation For validation of task implementations.
#449 by StellaAthena was closed Nov 8, 2023
@StellaAthena
Clarify Lambada Task bug Something isn't working. documentation Improvements or additions to documentation. help wanted Contributors and extra help welcome.
#356 by StellaAthena was closed Nov 26, 2022
Issue with the coqa task
#311 by StellaAthena was closed Apr 28, 2022
4
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.