Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
170 Open 591 Closed
170 Open 591 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

wrong regular expression for exact match scoring
#1303 opened Jan 17, 2024 by Hannibal046
1
Is it possible to use BLEU with multiple references?
#1125 opened Dec 14, 2023 by juliafalcao
2
[Discussion] Add Major Code Benchmarks opinions wanted For discussing open questions.
#1157 opened Dec 18, 2023 by haileyschoelkopf
6 tasks
1
4
Revamp + automate Task Table documentation feature documentation Improvements or additions to documentation. feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1160 opened Dec 18, 2023 by haileyschoelkopf
3 tasks
1
3
process_results() operating on the dataset, not example, level feature request A feature that isn't implemented yet. good first issue Good for newcomers
#1163 opened Dec 18, 2023 by StellaAthena
Organize / Cleanup Logging + Levels documentation Improvements or additions to documentation. feature request A feature that isn't implemented yet.
#1192 opened Dec 21, 2023 by haileyschoelkopf
1
Documentation about tasks <> request types mapping
#1202 opened Dec 22, 2023 by anjor
1
CoQA's implementation only predicts the last answer of each text bug Something isn't working. good first issue Good for newcomers
#1231 opened Jan 1, 2024 by glerzing
1
Implement XQuAD feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#191 opened Jun 10, 2021 by sdtblck
3
always get acc,acc_norm, perplexity =1 on triviaqa task based on llama2 model bug Something isn't working.
#1239 opened Jan 3, 2024 by learner-crapy
1
4
Speed up + streamline prompt template rendering runtime feature request A feature that isn't implemented yet. help wanted Contributors and extra help welcome.
#1286 opened Jan 15, 2024 by haileyschoelkopf
2
Low results on TriviaQA
#1292 opened Jan 16, 2024 by yafuly
7
KeyError on some metrics from huggingface/evaluate bug Something isn't working.
#1302 opened Jan 17, 2024 by alexrs
2
Quac Dataset feature request A feature that isn't implemented yet. good first issue Good for newcomers
#827 opened Sep 4, 2023 by RanchiZhao
8
Different fewshot with and w/o accelerate DDP
#1308 opened Jan 17, 2024 by baberabb
1
NAN value for truthfulqa_mc2 on full finetuned model TinyLlama
#1340 opened Jan 23, 2024 by hahmad2008
10
Hello, I would like to know if there is a method to use "generate_until" to evaluate on the ceval or cmmlu dataset. I'm using a chat model, which adds a prompt template to make it answer questions. However, the model's answer choices (like A, B, C, D) may not necessarily be the first generated token.
#1362 opened Jan 27, 2024 by noforit
1
How to compute the perplexity only on the answer? asking questions For asking for clarification / support on library usage.
#1370 opened Jan 30, 2024 by Luobots
7
Only a single filtered_resps is logged for repeat > 1 for each sample bug Something isn't working.
#1232 opened Jan 1, 2024 by baberabb
4
Implement MLQA feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#192 opened Jun 10, 2021 by sdtblck
Implement TyDiQA feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#193 opened Jun 10, 2021 by sdtblck
MNLI task giving (very) different results than the HuggingFace task accuracy metric bug Something isn't working. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#320 opened May 8, 2022 by JunShern
Central repository for results from running the evaluations
#662 opened Jul 6, 2023 by c1505
7
Issue when Evaluating ChatGLM
#705 opened Jul 26, 2023 by Tardissss
8
truthfulqa_mc2 is Nan, while truthfulqa_mc1 is 1.00
#714 opened Jul 31, 2023 by chi2liu
5
ProTip! Follow long discussions with comments:>50.