Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

214 Open 727 Closed
214 Open 727 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

AssertionError: aggregation named 'mean' conflicts with existing registered aggregation!
#1839 opened May 14, 2024 by hunter2009pf
Bug: wrong until default value for chat based model
#1837 opened May 14, 2024 by YilunZhou
2
Evaluation results of llama2 with lm-evaluation-harness using wikitext-2
#1833 opened May 13, 2024 by l2002924700
1
Using Language Models as Evaluators feature request A feature that isn't implemented yet.
#1831 opened May 13, 2024 by lintangsutawika
6
eval gsm8k from local dataset folder with the bug info "ValueError: BuilderConfig 'main' not found."
#1829 opened May 12, 2024 by Jp-17
@haileyschoelkopf
Add More Tests feature request A feature that isn't implemented yet.
#1827 opened May 12, 2024 by haileyschoelkopf
Avoid slow testing due to network issues.
#1824 opened May 11, 2024 by pixeli99
2
The input format for XNLI seems wired?
#1822 opened May 10, 2024 by SefaZeng
@lintangsutawika
2
TypeError: 'NoneType' object is not iterable when using cache and loglikelihood_rolling
#1821 opened May 10, 2024 by mdocekal
@haileyschoelkopf
Task description newline characters removed by Jinja templating, affecting generated requests and performance
#1817 opened May 9, 2024 by ma0li
2
Multi-round evaluation for chat models
#1816 opened May 9, 2024 by YilunZhou
1
Multi Label Classification
#1814 opened May 9, 2024 by IsraelAbebe
how to run all the bigbench tasks at once?
#1809 opened May 8, 2024 by kbmlcoding
Gemini 1.5/Ultra support
#1808 opened May 8, 2024 by notrichardren
Support loading slices of a split from a dataset
#1788 opened May 6, 2024 by alexrs
openai.InternalServerError: the model generated invalid Unicode output
#1783 opened May 4, 2024 by djstrong
How to filter to see only generate_until: lm-eval --tasks list
#1772 opened May 2, 2024 by chigkim
Support OpenAI's Batch API
#1770 opened May 2, 2024 by djstrong
1
IndexError: list index out of range when running benchmark on gguf model
#1768 opened Apr 30, 2024 by fherrmannsdoerfer
2
Cannot have both a group list and task list asking questions For asking for clarification / support on library usage. bug Something isn't working.
#1767 opened Apr 29, 2024 by steven-basart
5
Bug in yaml parsing
#1762 opened Apr 28, 2024 by jordane95
Does this support the model to use generate functions to eval not likelihood?
#1761 opened Apr 28, 2024 by Juhywcy
Output constrained support
#1759 opened Apr 27, 2024 by Mihaiii
HellaSwag with UnicodeDecodeError
#1757 opened Apr 27, 2024 by Hua-rookie
13
New Task Request: LegalBench feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1754 opened Apr 26, 2024 by haileyschoelkopf
2
ProTip! Follow long discussions with comments:>50.