Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
209 Open 686 Closed
209 Open 686 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Inconsistent evaluation results with Chat Template
#1841 opened May 14, 2024 by shiweijiezero
👍 8
Add support to azure-openai deployed models
#1733 opened Apr 22, 2024 by bcarvalho-via
2 👍 6
Integrate Semantic Answer Similarity (SAS) into the evaluation metrics.
#1703 opened Apr 15, 2024 by gonzalo-santamaria-iic
2 👍 6
Avoid slow testing due to network issues.
#1824 opened May 11, 2024 by pixeli99
2 👍 4
ValueError: BuilderConfig 'pile_freelaw' not found., issue on running PILE eval
#1714 opened Apr 16, 2024 by Harryalways317
👍 3
Should num_fewshot be type list? feature request A feature that isn't implemented yet.
#837 opened Sep 6, 2023 by Wehzie
1 👍 3
How to interpret generated results for truthful_qa test
#993 opened Nov 15, 2023 by Joetib
3 👍 3
How to filter to see only generate_until: lm-eval --tasks list
#1772 opened May 2, 2024 by chigkim
👍 2
Allow --include_path to import an externally-defined LM subclass feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1457 opened Feb 22, 2024 by haileyschoelkopf
1 👍 2
Save fewshot_as_multiturn argument in results.json
#1941 opened Jun 10, 2024 by djstrong
👍 1
Sanity checking the semantic meaning of "perplexity" in code asking questions For asking for clarification / support on library usage.
#1581 opened Mar 15, 2024 by RylanSchaeffer
1 👍 1
Whitespace before label in MultipleChoiceTask causes wrong label probability prediction
#1556 opened Mar 11, 2024 by RibinMTC
👍 1
Add a way to instantiate from HF.AutoModel (again)
#1978 opened Jun 17, 2024 by dmitrii-palisaderesearch
👍 1
Quac Dataset feature request A feature that isn't implemented yet. good first issue Good for newcomers
#827 opened Sep 4, 2023 by RanchiZhao
8 👍 1
OpenaiCompletionsLM invokes the completions API with max_tokens set to 0
#1903 opened May 29, 2024 by chimezie
1
1 👍 1
[TruthfulQA] update rouge-score version or add a way to suppress tokenizer logging
#1692 opened Apr 9, 2024 by skramer-dev
2 👍 1
Support loading slices of a split from a dataset
#1788 opened May 6, 2024 by alexrs
👍 1
Allow Task objects to defer dataset download feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1558 opened Mar 11, 2024 by haileyschoelkopf
2 👍 1
Gemini 1.5/Ultra support
#1808 opened May 8, 2024 by notrichardren
👍 1
[New Task] CommonsenseQA feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1026 opened Nov 27, 2023 by haileyschoelkopf
@StellaAthena
4 👍 1
Request for files to be placed in 'path/containing/training/set/ngrams'.
#1375 opened Jan 31, 2024 by dsdanielpark
👍 1
OpenAI models not working with truthfulqa_* tasks
#1704 opened Apr 15, 2024 by ichitaka
6 👍 1
Add tasks for performance on long context lengths feature request A feature that isn't implemented yet.
#1748 opened Apr 25, 2024 by nairbv
1 👍 1
wrong regular expression for exact match scoring
#1303 opened Jan 17, 2024 by Hannibal046
1 👍 1
globally normalized models
#960 opened Nov 3, 2023 by denizyuret
1 👍 1
ProTip! Find all open issues with in progress development work with linked:pr.