Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
205 Open 693 Closed
205 Open 693 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

AssertionError: aggregation named 'mean' conflicts with existing registered aggregation!
#1839 opened May 14, 2024 by hunter2009pf
Inconsistent evaluation results with Chat Template
#1841 opened May 14, 2024 by shiweijiezero
Llama2-hf-q40.gguf model got very poor results on lambada_openai tasks, but was fine on other tasks.
#1866 opened May 21, 2024 by intellinjun
chat model evaluation
#1870 opened May 22, 2024 by jordane95
TypeError of scrolls_narrativeqa
#1891 opened May 27, 2024 by hicleo
@lintangsutawika
model_comparator.py broken
#1899 opened May 29, 2024 by johnwee1
--hf_hub_log_args causes IndexError
#1925 opened Jun 5, 2024 by johnwee1
High Number of Tokens for openai-completions Models
#1936 opened Jun 7, 2024 by selinaxiao
Regarding decontamination
#1938 opened Jun 9, 2024 by dsdanielpark
Add MMLU-Pro Dataset feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1947 opened Jun 11, 2024 by haileyschoelkopf
1
Check compatibility of local-completions with VLLM (returns logits) for multiple_choice tasks bug Something isn't working.
#1949 opened Jun 11, 2024 by haileyschoelkopf
Error when chat template is not a string
#1962 opened Jun 13, 2024 by djstrong
Incorrect Multilingual arc implementation
#2000 opened Jun 19, 2024 by hynky1999
Add HuggingFace Text-Generation-Interface Support
#2001 opened Jun 19, 2024 by taoari
Implementing lessons from OLMES
#2002 opened Jun 20, 2024 by lintangsutawika
Should num_fewshot be type list? feature request A feature that isn't implemented yet.
#837 opened Sep 6, 2023 by Wehzie
1
globally normalized models
#960 opened Nov 3, 2023 by denizyuret
1
Stability Upstream translated task feature request A feature that isn't implemented yet.
#1006 opened Nov 20, 2023 by StellaAthena
@StellaAthena
1
Add task variants replicating Llama 1 / 2 evaluation numbers feature request A feature that isn't implemented yet.
#1078 opened Dec 7, 2023 by haileyschoelkopf
@haileyschoelkopf
1
Organize / Cleanup Logging + Levels documentation Improvements or additions to documentation. feature request A feature that isn't implemented yet.
#1192 opened Dec 21, 2023 by haileyschoelkopf
1
Documentation about tasks <> request types mapping
#1202 opened Dec 22, 2023 by anjor
1
CoQA's implementation only predicts the last answer of each text bug Something isn't working. good first issue Good for newcomers
#1231 opened Jan 1, 2024 by glerzing
1
wrong regular expression for exact match scoring
#1303 opened Jan 17, 2024 by Hannibal046
1
Different fewshot with and w/o accelerate DDP
#1308 opened Jan 17, 2024 by baberabb
1
Hello, I would like to know if there is a method to use "generate_until" to evaluate on the ceval or cmmlu dataset. I'm using a chat model, which adds a prompt template to make it answer questions. However, the model's answer choices (like A, B, C, D) may not necessarily be the first generated token.
#1362 opened Jan 27, 2024 by noforit
1
ProTip! no:milestone will show everything without a milestone.