Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
33 Open 23 Closed
33 Open 23 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

--cache_requests sometimes errors when run with accelerate launch bug Something isn't working.
#1564 by haileyschoelkopf was closed Mar 18, 2024 v0.4.2 Release
1
Run Gemma LM in Huggingface (simple patch) feature request A feature that isn't implemented yet.
#1455 by haileyschoelkopf was closed Feb 26, 2024
2 tasks
1
Display n-shot better for groups and for hardcoded-fewshot tasks feature request A feature that isn't implemented yet.
#1360 by haileyschoelkopf was closed Feb 1, 2024
1
[New Task] Upstream remaining Okapi multilingual tasks feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1244 by haileyschoelkopf was closed Feb 21, 2024
8
Add Logits to OpenAI ChatCompletions model declined A proposed dataset or feature request that will not be implemented. feature request A feature that isn't implemented yet. help wanted Contributors and extra help welcome.
#1196 by haileyschoelkopf was closed May 23, 2024
29
Supplied process_docs() function is not applied to fewshot docs bug Something isn't working.
#1179 by haileyschoelkopf was closed Jan 12, 2024
1
@haileyschoelkopf
[New Task] Paloma Eval Suite feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1176 by haileyschoelkopf was closed Jun 19, 2024
1
@lintangsutawika
Unify "metric" and "aggregation" abstractions feature request A feature that isn't implemented yet.
#1158 by haileyschoelkopf was closed Mar 15, 2024
@lintangsutawika
1
Print "higher_is_better" in results table feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1153 by haileyschoelkopf was closed Jun 3, 2024
1
2
Add --predict_only mode (run without scoring outputs) feature request A feature that isn't implemented yet. help wanted Contributors and extra help welcome.
#1152 by haileyschoelkopf was closed Jan 31, 2024
1
17
Support wrapping prompts with a given Chat Template feature request A feature that isn't implemented yet. help wanted Contributors and extra help welcome. opinions wanted For discussing open questions.
#1098 by haileyschoelkopf was closed Jun 11, 2024 v0.4.3
@haileyschoelkopf
27
Upstream Mamba integration feature request A feature that isn't implemented yet.
#1085 by haileyschoelkopf was closed Dec 22, 2023
1
@haileyschoelkopf
2
[New Task] SIQA feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1027 by haileyschoelkopf was closed Nov 28, 2023
@StellaAthena
[New Task Request] IFEval / Instruction-Following Eval feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1012 by haileyschoelkopf was closed Feb 9, 2024
[New Task] Implement GPQA dataset feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1010 by haileyschoelkopf was closed Mar 5, 2024
1
1
[Refactor] Remove gold_alias stale YAML config option
#964 by haileyschoelkopf was closed Nov 6, 2023 v0.4.0
1
@haileyschoelkopf
[Refactor] Incorporate version field for tasks into metadata feature request A feature that isn't implemented yet.
#963 by haileyschoelkopf was closed Nov 29, 2023 v0.4.0
1
@lintangsutawika
[Refactor] Allow for some tasks to force zero-shot feature request A feature that isn't implemented yet.
#962 by haileyschoelkopf was closed Nov 29, 2023 v0.4.0
1
@lintangsutawika
1
[Refactor] Revamp Testing / CI pipeline feature request A feature that isn't implemented yet. help wanted Contributors and extra help welcome.
#656 by haileyschoelkopf was closed Mar 4, 2024
2 of 6 tasks
1
[Refactor] Collecting Feedback + Bug reports
#592 by haileyschoelkopf was closed Aug 8, 2023
1
Temperature Sampling + Maj@K
#387 by haileyschoelkopf was closed Feb 9, 2023
Add Anthropic Model-written Eval datasets to harness? feature request A feature that isn't implemented yet. good first issue Good for newcomers
#375 by haileyschoelkopf was closed Nov 8, 2023
2 tasks
1
6
Add Big-Bench Lite to eval-harness
#345 by haileyschoelkopf was closed Apr 24, 2023
@lintangsutawika @haileyschoelkopf
3
ProTip! Follow long discussions with comments:>50.