-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Issues: EleutherAI/lm-evaluation-harness
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
lm_eval --model vllm did not work when data_parallel_size > 1
bug
Something isn't working.
#2379
opened Oct 3, 2024 by
wukaixingxp
Is LLaMA3.2-Vision-90B/11B result on mmmu_val reproducible?
validation
For validation of task implementations.
#2377
opened Oct 2, 2024 by
jybbjybb
Which filter value should be used among the accuracy test results?
#2362
opened Sep 27, 2024 by
KKwanhee
[multimodal] llava-1.5-7b-hf doesn't work on Something isn't working.
mmmu_val
bug
#2360
opened Sep 26, 2024 by
BabyChouSr
Improve Improvements or additions to documentation.
feature request
A feature that isn't implemented yet.
docs/model_guide.md
with skeleton template code + description of utils like Collator
and Reorderer
documentation
#2358
opened Sep 26, 2024 by
haileyschoelkopf
Add a test for
scripts/write_out.py
and other scripts/
utils
#2356
opened Sep 26, 2024 by
haileyschoelkopf
Setting limit_mm_per_prompt for vllm_vlm fails argument parser
bug
Something isn't working.
#2352
opened Sep 25, 2024 by
mgoin
The base model and chat model have no difference when using generate_until, loglikelihood, loglikelihood_rolling,right?
asking questions
For asking for clarification / support on library usage.
#2347
opened Sep 25, 2024 by
belle9217
Reproduce QWen 2.5-14B-Instruct and LLaMa-3.1-8B-Instruct Results
#2344
opened Sep 25, 2024 by
ruleGreen
Locally reproducible HF-Leaderboard evals
asking questions
For asking for clarification / support on library usage.
#2338
opened Sep 24, 2024 by
eldarkurtic
Dynamical prompt with extremely promising results #RIPrompt
#2335
opened Sep 23, 2024 by
anthonyrisinger
Error for AGIEval when using fewshot
bug
Something isn't working.
validation
For validation of task implementations.
#2323
opened Sep 19, 2024 by
BaohaoLiao
Which version to use
validation
For validation of task implementations.
#2322
opened Sep 19, 2024 by
sorobedio
Previous Next
ProTip!
Follow long discussions with comments:>50.