Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
123 Open 137 Closed
123 Open 137 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Add support to azure-openai deployed models
#1733 opened Apr 22, 2024 by bcarvalho-via
3 👍 9
Inconsistent evaluation results with Chat Template
#1841 opened May 14, 2024 by shiweijiezero
4 👍 9
Add tasks for performance on long context lengths feature request A feature that isn't implemented yet.
#1748 opened Apr 25, 2024 by nairbv
2 👍 4
Having issues with MMLU benchmark
#2094 opened Jul 12, 2024 by berkatil
@haileyschoelkopf
9 👍 3
ValueError: BuilderConfig 'pile_freelaw' not found., issue on running PILE eval
#1714 opened Apr 16, 2024 by Harryalways317
@haileyschoelkopf
2 👍 3
How to RUN BENCHMARK on GGUF Models ?
#2086 opened Jul 10, 2024 by RakshitAralimatti
👍 2
Add long context evaluation benchmarks such as LongBench and LEval. feature request A feature that isn't implemented yet. help wanted Contributors and extra help welcome.
#2180 opened Aug 5, 2024 by txchen-USTC
2 👍 1
Allow Task objects to defer dataset download feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1558 opened Mar 11, 2024 by haileyschoelkopf
3 👍 1
Bug in the visualize-wandb.ipynb example
#2147 opened Jul 29, 2024 by tanaymeh
👍 1
IFEval fails when multiple gpus are used (for DDP)
#2266 opened Aug 30, 2024 by al093
1
7
Dynamical prompt with extremely promising results #RIPrompt
#2335 opened Sep 23, 2024 by anthonyrisinger
1
Error for AGIEval when using fewshot bug Something isn't working. validation For validation of task implementations.
#2323 opened Sep 19, 2024 by BaohaoLiao
1
Which version to use validation For validation of task implementations.
#2322 opened Sep 19, 2024 by sorobedio
9
Confusion over the model outputs
#2331 opened Sep 23, 2024 by tranlm
Failed to add a new metric
#2330 opened Sep 23, 2024 by Ofir408
eval gsm8k from local dataset folder with the bug info "ValueError: BuilderConfig 'main' not found."
#1829 opened May 12, 2024 by Jp-17
@haileyschoelkopf
4
Locally reproducible HF-Leaderboard evals asking questions For asking for clarification / support on library usage.
#2338 opened Sep 24, 2024 by eldarkurtic
2
New Task: openai_mmmlu professionaly translated by OpenAI as part of o1 release feature request A feature that isn't implemented yet.
#2305 opened Sep 16, 2024 by giuliolovisotto
1
gpt2 evaluation
#2343 opened Sep 24, 2024 by sorobedio
toxigen task measures toxicity classification rather than whether generations are toxic?
#974 opened Nov 8, 2023 by laphang
8
tasks RACE only high not "middle"
#2345 opened Sep 25, 2024 by Choi-jun9803
Low results on TriviaQA
#1292 opened Jan 16, 2024 by yafuly
13
Truncation asking questions For asking for clarification / support on library usage.
#1426 opened Feb 13, 2024 by mdocekal
4
evaluation extremely slow with llama_cpp/gguf bug Something isn't working.
#1472 opened Feb 26, 2024 by mobicham
@baberabb
5
coqa not working bug Something isn't working.
#1529 opened Mar 5, 2024 by lchu-ibm
@haileyschoelkopf
2
ProTip! Mix and match filters to narrow down what you’re looking for.