Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

206 Open 694 Closed
206 Open 694 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

how to run all the bigbench tasks at once?
#1809 opened May 8, 2024 by kbmlcoding
Gemini 1.5/Ultra support
#1808 opened May 8, 2024 by notrichardren
Support loading slices of a split from a dataset
#1788 opened May 6, 2024 by alexrs
openai.InternalServerError: the model generated invalid Unicode output
#1783 opened May 4, 2024 by djstrong
How to filter to see only generate_until: lm-eval --tasks list
#1772 opened May 2, 2024 by chigkim
Support OpenAI's Batch API
#1770 opened May 2, 2024 by djstrong
1
IndexError: list index out of range when running benchmark on gguf model
#1768 opened Apr 30, 2024 by fherrmannsdoerfer
2
Cannot have both a group list and task list asking questions For asking for clarification / support on library usage. bug Something isn't working.
#1767 opened Apr 29, 2024 by steven-basart
5
Bug in yaml parsing
#1762 opened Apr 28, 2024 by jordane95
Does this support the model to use generate functions to eval not likelihood?
#1761 opened Apr 28, 2024 by Juhywcy
Output constrained support
#1759 opened Apr 27, 2024 by Mihaiii
HellaSwag with UnicodeDecodeError
#1757 opened Apr 27, 2024 by Hua-rookie
11
New Task Request: LegalBench feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1754 opened Apr 26, 2024 by haileyschoelkopf
2
Accuracy gap between single GPU and multiple GPUs
#1751 opened Apr 26, 2024 by HsuWanTing
3
Add tasks for performance on long context lengths feature request A feature that isn't implemented yet.
#1748 opened Apr 25, 2024 by nairbv
1
[Feature Request] Metrics that require knowledge of input.
#1747 opened Apr 24, 2024 by ciaranby
[Feature Request] pre-built Docker image support feature request A feature that isn't implemented yet. help wanted Contributors and extra help welcome.
#1746 opened Apr 24, 2024 by zsaladin
3
accelerate doesn't work with auto:(>1)
#1743 opened Apr 24, 2024 by ozgurcelik
4
Process hangs when using tensor_parallel_size and data_parallel_size together bug Something isn't working.
#1734 opened Apr 22, 2024 by harshakokel
8
Add support to azure-openai deployed models
#1733 opened Apr 22, 2024 by bcarvalho-via
3
How to change ceval task to eval ceval dataset few-shot, cot-zero-shot, cot-few-show?
#1720 opened Apr 18, 2024 by lemon-little
When using Accelerate for data parallel inference, using different numbers of GPUs results in different results
#1719 opened Apr 18, 2024 by s1ghhh
4
Single doc results of model inference not found
#1718 opened Apr 18, 2024 by EugeneYCSong1212
MMLU output_type from multiple_choice to generate_until
#1716 opened Apr 17, 2024 by colourful-tree
ValueError: BuilderConfig 'pile_freelaw' not found., issue on running PILE eval
#1714 opened Apr 16, 2024 by Harryalways317
ProTip! Mix and match filters to narrow down what you’re looking for.