Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

214 Open 727 Closed
214 Open 727 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Accuracy gap between single GPU and multiple GPUs
#1751 opened Apr 26, 2024 by HsuWanTing
3
Add tasks for performance on long context lengths feature request A feature that isn't implemented yet.
#1748 opened Apr 25, 2024 by nairbv
1
[Feature Request] Metrics that require knowledge of input.
#1747 opened Apr 24, 2024 by ciaranby
[Feature Request] pre-built Docker image support feature request A feature that isn't implemented yet. help wanted Contributors and extra help welcome.
#1746 opened Apr 24, 2024 by zsaladin
3
accelerate doesn't work with auto:(>1)
#1743 opened Apr 24, 2024 by ozgurcelik
4
Process hangs when using tensor_parallel_size and data_parallel_size together bug Something isn't working.
#1734 opened Apr 22, 2024 by harshakokel
8
Add support to azure-openai deployed models
#1733 opened Apr 22, 2024 by bcarvalho-via
3
How to change ceval task to eval ceval dataset few-shot, cot-zero-shot, cot-few-show?
#1720 opened Apr 18, 2024 by lemon-little
When using Accelerate for data parallel inference, using different numbers of GPUs results in different results
#1719 opened Apr 18, 2024 by s1ghhh
4
Single doc results of model inference not found
#1718 opened Apr 18, 2024 by EugeneYCSong1212
MMLU output_type from multiple_choice to generate_until
#1716 opened Apr 17, 2024 by colourful-tree
ValueError: BuilderConfig 'pile_freelaw' not found., issue on running PILE eval
#1714 opened Apr 16, 2024 by Harryalways317
Passing a config file to wandb using CLI
#1713 opened Apr 16, 2024 by ozgurcelik
OpenAI models not working with truthfulqa_* tasks
#1704 opened Apr 15, 2024 by ichitaka
6
Integrate Semantic Answer Similarity (SAS) into the evaluation metrics.
#1703 opened Apr 15, 2024 by gonzalo-santamaria-iic
2
GSM8K Error result produced by gpt-3.5-turbo
#1700 opened Apr 14, 2024 by APiaoG
Better Document Available Tasks
#1698 opened Apr 12, 2024 by Harryalways317
@Harryalways317
14
Support for other languages - Translated datasets
#1695 opened Apr 11, 2024 by onurgu
IndexError on BBH tasks
#1688 opened Apr 8, 2024 by RylanSchaeffer
2
Better Document Data-Parallel interface / clean it up feature request A feature that isn't implemented yet.
#1684 opened Apr 7, 2024 by haileyschoelkopf
Cleanup Dependencies Further feature request A feature that isn't implemented yet.
#1683 opened Apr 7, 2024 by haileyschoelkopf
Add docstring for HFLM's many keyword args documentation Improvements or additions to documentation. feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1682 opened Apr 7, 2024 by haileyschoelkopf
Add nemo LM class to table of supported models / libraries documentation Improvements or additions to documentation.
#1681 opened Apr 7, 2024 by haileyschoelkopf
Batch size auto OOM bug Something isn't working.
#1678 opened Apr 6, 2024 by sam-paech
5
List of until tokens in cli args not supported feature request A feature that isn't implemented yet.
#1677 opened Apr 5, 2024 by tianyi-chen
@haileyschoelkopf
3
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.