Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
205 Open 693 Closed
205 Open 693 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

id_column contains duplicate values
#1667 opened Apr 3, 2024 by nicho2
Add nemo LM class to table of supported models / libraries documentation Improvements or additions to documentation.
#1681 opened Apr 7, 2024 by haileyschoelkopf
Add docstring for HFLM's many keyword args documentation Improvements or additions to documentation. feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1682 opened Apr 7, 2024 by haileyschoelkopf
Cleanup Dependencies Further feature request A feature that isn't implemented yet.
#1683 opened Apr 7, 2024 by haileyschoelkopf
Better Document Data-Parallel interface / clean it up feature request A feature that isn't implemented yet.
#1684 opened Apr 7, 2024 by haileyschoelkopf
Support for other languages - Translated datasets
#1695 opened Apr 11, 2024 by onurgu
GSM8K Error result produced by gpt-3.5-turbo
#1700 opened Apr 14, 2024 by APiaoG
Passing a config file to wandb using CLI
#1713 opened Apr 16, 2024 by ozgurcelik
ValueError: BuilderConfig 'pile_freelaw' not found., issue on running PILE eval
#1714 opened Apr 16, 2024 by Harryalways317
MMLU output_type from multiple_choice to generate_until
#1716 opened Apr 17, 2024 by colourful-tree
Single doc results of model inference not found
#1718 opened Apr 18, 2024 by EugeneYCSong1212
How to change ceval task to eval ceval dataset few-shot, cot-zero-shot, cot-few-show?
#1720 opened Apr 18, 2024 by lemon-little
[Feature Request] Metrics that require knowledge of input.
#1747 opened Apr 24, 2024 by ciaranby
Output constrained support
#1759 opened Apr 27, 2024 by Mihaiii
Does this support the model to use generate functions to eval not likelihood?
#1761 opened Apr 28, 2024 by Juhywcy
Bug in yaml parsing
#1762 opened Apr 28, 2024 by jordane95
How to filter to see only generate_until: lm-eval --tasks list
#1772 opened May 2, 2024 by chigkim
openai.InternalServerError: the model generated invalid Unicode output
#1783 opened May 4, 2024 by djstrong
Support loading slices of a split from a dataset
#1788 opened May 6, 2024 by alexrs
Gemini 1.5/Ultra support
#1808 opened May 8, 2024 by notrichardren
how to run all the bigbench tasks at once?
#1809 opened May 8, 2024 by kbmlcoding
Multi Label Classification
#1814 opened May 9, 2024 by IsraelAbebe
TypeError: 'NoneType' object is not iterable when using cache and loglikelihood_rolling
#1821 opened May 10, 2024 by mdocekal
@haileyschoelkopf
Add More Tests feature request A feature that isn't implemented yet.
#1827 opened May 12, 2024 by haileyschoelkopf
eval gsm8k from local dataset folder with the bug info "ValueError: BuilderConfig 'main' not found."
#1829 opened May 12, 2024 by Jp-17
@haileyschoelkopf
ProTip! Mix and match filters to narrow down what you’re looking for.