Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
3 Open 13 Closed
3 Open 13 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Add Anthropic Model-written Eval datasets to harness? feature request A feature that isn't implemented yet. good first issue Good for newcomers
#375 by haileyschoelkopf was closed Nov 8, 2023
2 tasks
1
6
[Refactor] Incorporate version field for tasks into metadata feature request A feature that isn't implemented yet.
#963 by haileyschoelkopf was closed Nov 29, 2023 v0.4.0
1
@lintangsutawika
[Refactor] Allow for some tasks to force zero-shot feature request A feature that isn't implemented yet.
#962 by haileyschoelkopf was closed Nov 29, 2023 v0.4.0
1
@lintangsutawika
1
[Refactor] Remove gold_alias stale YAML config option
#964 by haileyschoelkopf was closed Nov 6, 2023 v0.4.0
1
@haileyschoelkopf
Supplied process_docs() function is not applied to fewshot docs bug Something isn't working.
#1179 by haileyschoelkopf was closed Jan 12, 2024
1
@haileyschoelkopf
[New Task] Implement GPQA dataset feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1010 by haileyschoelkopf was closed Mar 5, 2024
1
1
Run Gemma LM in Huggingface (simple patch) feature request A feature that isn't implemented yet.
#1455 by haileyschoelkopf was closed Feb 26, 2024
2 tasks
1
Add --predict_only mode (run without scoring outputs) feature request A feature that isn't implemented yet. help wanted Contributors and extra help welcome.
#1152 by haileyschoelkopf was closed Jan 31, 2024
1
17
Display n-shot better for groups and for hardcoded-fewshot tasks feature request A feature that isn't implemented yet.
#1360 by haileyschoelkopf was closed Feb 1, 2024
1
--cache_requests sometimes errors when run with accelerate launch bug Something isn't working.
#1564 by haileyschoelkopf was closed Mar 18, 2024 v0.4.2 Release
1
Upstream Mamba integration feature request A feature that isn't implemented yet.
#1085 by haileyschoelkopf was closed Dec 22, 2023
1
@haileyschoelkopf
2
Print "higher_is_better" in results table feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1153 by haileyschoelkopf was closed Jun 3, 2024
1
2
[New Task] Paloma Eval Suite feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1176 by haileyschoelkopf was closed Jun 19, 2024
1
@lintangsutawika
ProTip! Adding no:label will show everything without a label.