Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
261 Open 758 Closed
261 Open 758 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Multi-device evaluation with data parallel
#801 by YihanCao123 was closed Oct 14, 2023
3 👀 2
[FEATURE REQUEST] Can we have HumanEval+ benchmark? feature request A feature that isn't implemented yet.
#1091 by hahuyhoang411 was closed Dec 18, 2023
1 👀 2
Large discrepancy between Open LLM Leaderboard and running eval here
#1101 by l3utterfly was closed Dec 12, 2023
4 👀 2
Tasks on code evaluation
#1282 by yifan-bao was closed Jan 18, 2024
1 👀 2
Incorrect evaluation results when using 'mps' backend
#1679 by zifeitong was closed Apr 8, 2024
1
2 👀 1
Different score when using accelerate bug Something isn't working.
#1293 by lintangsutawika was closed Jan 25, 2024
1
4 👀 1
Reported metrics are different with multi-node
#1081 by keeganq was closed Feb 1, 2024
1
4 👀 1
Prompt Templating
#896 by sachith-surge was closed Oct 10, 2023
6 👀 1
How can i evaluate the local model
#1568 by qxpBlog was closed Mar 13, 2024
5 👀 1
Getting different results on standard benchmarks for Llama-2 7B
#805 by adil-a was closed Nov 8, 2023
6 👀 1
TypeError: Union[arg, ...]: each arg must be a type. Got True.
#530 by passaglia was closed May 30, 2023
1 👀 1
[big refactor]mmlu is needed
#875 by xiaol was closed Oct 10, 2023
1 👀 1
[New Feature] Addressing Data Contamination in Evaluation Benchmarks
#1056 by liyucheng09 was closed Dec 30, 2023
2 👀 1
I get this error whenever I try to run an eval: ImportError: cannot import name 'HfApi' from 'huggingface_hub'
#1826 by menhguin was closed May 26, 2024
1
@haileyschoelkopf
15 👀 1
dataset path asking questions For asking for clarification / support on library usage.
#1659 by ghost was closed Apr 2, 2024
9 👀 1
Include support for evaluation of the GPT-Neo Models
#197 by mihaela-bornea was closed Aug 26, 2021
5 👀 1
Implement the English Grammar Correction evaluation feature request A feature that isn't implemented yet. good first issue Good for newcomers
#28 by StellaAthena was closed Nov 21, 2022
2 tasks
Implement the News Article Generation evaluation feature request A feature that isn't implemented yet.
#29 by StellaAthena was closed Nov 21, 2022
2 tasks
IndexError: index out of bounds for multiple tasks of mmlu_pro
#2240 by lxning was closed Aug 22, 2024
1
Implement the QuAC evaluation feature request A feature that isn't implemented yet. good first issue Good for newcomers
#18 by StellaAthena was closed Nov 14, 2023
1 of 2 tasks
1
6
Implement the CoQA evaluation feature request A feature that isn't implemented yet. good first issue Good for newcomers
#17 by StellaAthena was closed Feb 14, 2021
1 of 2 tasks
2
@thefazzer
1
Implement the OpenBookQA evaluation feature request A feature that isn't implemented yet. good first issue Good for newcomers
#16 by StellaAthena was closed Feb 9, 2021
2 tasks done
1
@cfoster0 @jon-tow
3
Implement the ARC Challenge evaluation feature request A feature that isn't implemented yet. good first issue Good for newcomers
#15 by StellaAthena was closed Feb 5, 2021
2 tasks done
@jon-tow
1
Implement the WebQuestions evaluation feature request A feature that isn't implemented yet. good first issue Good for newcomers
#10 by StellaAthena was closed Feb 8, 2021
1 of 2 tasks
@jon-tow
1
Implement the Natural Questions evaluation feature request A feature that isn't implemented yet. good first issue Good for newcomers
#9 by StellaAthena was closed Aug 21, 2023
1 of 2 tasks
1
19
ProTip! Type g i on any issue or pull request to go back to the issue listing page.