Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
282 Open 807 Closed
282 Open 807 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Error when chat template is not a string
#1962 opened Jun 13, 2024 by djstrong
1
How to use a vllm hosted model?
#1963 opened Jun 13, 2024 by darsh-essential
1
What is the output_type in the metric for?
#1976 opened Jun 17, 2024 by dennisrall
1
The evaluation results are inconsistent across different GPUs
#2077 opened Jul 8, 2024 by DonteFlynn
1
The response is too short to extract answer on GPQA. What should I set to extend it?
#2081 opened Jul 8, 2024 by URRealHero
1
Question: Realtoxicityprompts takes >10 seconds per query, is this expected behavior?
#2096 opened Jul 12, 2024 by meg-huggingface
1
Empty --log_samples outputs
#2115 opened Jul 19, 2024 by IsraelAbebe
1
ValueError when task name collide with local directory names
#2122 opened Jul 20, 2024 by alat-rights
@haileyschoelkopf
1
How to use Custom Prompt during Evaluation
#2131 opened Jul 23, 2024 by RakshitAralimatti
1
[Question] A way to run multiple evals on multiple models?
#2145 opened Jul 29, 2024 by tanaymeh
1
GPT2 eval in lambada_openai, acc only 0.325
#2159 opened Jul 30, 2024 by KeyKy
1
Premature num_fewshot check with fewshot_as_multiturn
#2161 opened Jul 30, 2024 by baberabb
1
1
How to calculate the "token_perplexity"
#2170 opened Aug 2, 2024 by nongfang55
1
1
local-completion: Passing Messages as List to Prompt Instead of Required String Format
#2171 opened Aug 2, 2024 by ankush13r
1
Invalid response for loglikelihood for GGUF model
#2213 opened Aug 15, 2024 by jaslatendresse
@baberabb
1
test_version_stable.py does not exist?
#2221 opened Aug 17, 2024 by dorsa-zeinali
1
Allow Prefilling Assistant Response w/ Chat Templates feature request A feature that isn't implemented yet.
#2248 opened Aug 23, 2024 by haileyschoelkopf
1
1
Regarding metric chrf's implementation feature request A feature that isn't implemented yet. good first issue Good for newcomers
#2256 opened Aug 28, 2024 by PinzhenChen
1
Bug in Leaderboard IFEval Code validation For validation of task implementations.
#2260 opened Aug 29, 2024 by noowad93
1
Worse evaluation performance with PEFT adaptors
#2294 opened Sep 13, 2024 by YananLi18
1
Low GPU Utilization During Multi-GPU evaluation - Efficiency Optimization asking questions For asking for clarification / support on library usage.
#2296 opened Sep 14, 2024 by yang3121099
1
New Task: openai_mmmlu professionaly translated by OpenAI as part of o1 release feature request A feature that isn't implemented yet.
#2305 opened Sep 16, 2024 by giuliolovisotto
1
Multiple generations (sequential) per question asking questions For asking for clarification / support on library usage. feature request A feature that isn't implemented yet.
#2317 opened Sep 17, 2024 by IntrepidEnki
1
Error for AGIEval when using fewshot bug Something isn't working. validation For validation of task implementations.
#2323 opened Sep 19, 2024 by BaohaoLiao
1
Dynamical prompt with extremely promising results #RIPrompt
#2335 opened Sep 23, 2024 by anthonyrisinger
1
ProTip! Add no:assignee to see everything that’s not assigned.