-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Issues: EleutherAI/lm-evaluation-harness
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
[FEATURE REQUEST] Can we have HumanEval+ benchmark?
feature request
A feature that isn't implemented yet.
#1091
by hahuyhoang411
was closed Dec 18, 2023
Large discrepancy between Open LLM Leaderboard and running eval here
#1101
by l3utterfly
was closed Dec 12, 2023
Different score when using Something isn't working.
accelerate
bug
#1293
by lintangsutawika
was closed Jan 25, 2024
Getting different results on standard benchmarks for Llama-2 7B
#805
by adil-a
was closed Nov 8, 2023
TypeError: Union[arg, ...]: each arg must be a type. Got True.
#530
by passaglia
was closed May 30, 2023
[New Feature] Addressing Data Contamination in Evaluation Benchmarks
#1056
by liyucheng09
was closed Dec 30, 2023
I get this error whenever I try to run an eval: ImportError: cannot import name 'HfApi' from 'huggingface_hub'
#1826
by menhguin
was closed May 26, 2024
dataset path
asking questions
For asking for clarification / support on library usage.
#1659
by ghost
was closed Apr 2, 2024
Implement the English Grammar Correction evaluation
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
#28
by StellaAthena
was closed Nov 21, 2022
2 tasks
Implement the News Article Generation evaluation
feature request
A feature that isn't implemented yet.
#29
by StellaAthena
was closed Nov 21, 2022
2 tasks
IndexError: index out of bounds for multiple tasks of mmlu_pro
#2240
by lxning
was closed Aug 22, 2024
Implement the QuAC evaluation
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
#18
by StellaAthena
was closed Nov 14, 2023
1 of 2 tasks
Implement the CoQA evaluation
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
#17
by StellaAthena
was closed Feb 14, 2021
1 of 2 tasks
Implement the OpenBookQA evaluation
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
#16
by StellaAthena
was closed Feb 9, 2021
2 tasks done
Implement the ARC Challenge evaluation
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
#15
by StellaAthena
was closed Feb 5, 2021
2 tasks done
Implement the WebQuestions evaluation
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
#10
by StellaAthena
was closed Feb 8, 2021
1 of 2 tasks
Implement the Natural Questions evaluation
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
#9
by StellaAthena
was closed Aug 21, 2023
1 of 2 tasks
Previous Next
ProTip!
Type g i on any issue or pull request to go back to the issue listing page.