Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
219 Open 727 Closed
219 Open 727 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

cannot eval wikitext2 bug Something isn't working.
#1665 by ghost was closed Apr 5, 2024
@haileyschoelkopf
6
Module 'exact_match' doesn't exist on the Hugging Face Hub either.
#1697 by BBaekdabang was closed Jul 1, 2024
1
@haileyschoelkopf
6
Results is weird for Qwen2-1.5B
#1944 by SefaZeng was closed Jun 24, 2024
6
not getting same accuracy as on the leaderboard when evaluating locally.
#2091 by sorobedio was closed Jul 13, 2024
6
Implement arithmetic evaluations feature request A feature that isn't implemented yet. good first issue Good for newcomers
#25 by StellaAthena was closed Jan 28, 2021
2 tasks done
@VitamintK
7
New Evaluation: Math feature request A feature that isn't implemented yet. good first issue Good for newcomers
#77 by StellaAthena was closed Feb 25, 2022
2 tasks
1
@hendrycks
7
Failure to evaluate facebook/opt-350m model bug Something isn't working. good first issue Good for newcomers
#368 by underactuated was closed Mar 9, 2023
7
Validate TriviaQA good first issue Good for newcomers validation For validation of task implementations.
#456 by StellaAthena was closed Jun 14, 2023
1
7
Can't read json file with pip install or git
#588 by SingL3 was closed Jul 5, 2023
7
[big-refactor] Accelerate launch FSDP Runtime Error
#892 by adamjackson2357 was closed Mar 4, 2024
7
AutoGPTQ not working with HuggingFace accelerate (multi GPU) bug Something isn't working.
#1247 by JeevanBhoot was closed Jan 15, 2024
1
@haileyschoelkopf
7
how to load a local pruned model
#1309 by qxpBlog was closed Jan 19, 2024
7
dataset path asking questions For asking for clarification / support on library usage.
#1659 by ghost was closed Apr 2, 2024
7
Out-Of-Memory Error for same batch size but different dataset
#1811 by richardzhuang0412 was closed May 9, 2024
7
New Evaluation: Biology feature request A feature that isn't implemented yet. good first issue Good for newcomers
#78 by StellaAthena was closed Nov 21, 2022
2 tasks
@nicholaskross
8
Custom data folder caching is relative instead of absolute
#247 by thomasw21 was closed Apr 1, 2022
@jon-tow
8
Validate MMMLU validation For validation of task implementations.
#475 by ollmer was closed Jun 23, 2023
1
@ollmer
8
How to deal with load_in_8bit error? bug Something isn't working.
#607 by ryusangwon was closed Aug 8, 2023
8
ConnectionError: Couldn't reach 'truthful_qa' on the Hub (ProxyError)
#762 by SefaZeng was closed Aug 11, 2023
8
[refactor] squadv2 task results quite different than main branch
#938 by emilyvanark was closed Nov 21, 2023
8
KeyError: 'mc1_targets' for truthfulqa_mc
#950 by umarbeknasimov was closed Dec 18, 2023
8
Same result on GPTQ 8bit and 4bit model, normal? bug Something isn't working.
#973 by Chrisz236 was closed Nov 22, 2023
8
Generator Error when evaluating GLUE and superGLUE bug Something isn't working.
#1240 by shiweijiezero was closed Jan 22, 2024
1
8
[New Task] Upstream remaining Okapi multilingual tasks feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1244 by haileyschoelkopf was closed Feb 21, 2024
8
mismatch result of triviaqa when using vllm and hf
#1262 by learner-crapy was closed Jan 11, 2024
1
8
ProTip! no:milestone will show everything without a milestone.