Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
205 Open 693 Closed
205 Open 693 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Batch size auto OOM bug Something isn't working.
#1678 opened Apr 6, 2024 by sqrkl
5
Cannot have both a group list and task list asking questions For asking for clarification / support on library usage. bug Something isn't working.
#1767 opened Apr 29, 2024 by steven-basart
5
"Please select a token to use as pad_token" error for alpaca-lora-7b model
#434 opened Apr 24, 2023 by oshev
@StellaAthena
6
OpenAI models not working with truthfulqa_* tasks
#1704 opened Apr 15, 2024 by ichitaka
6
Central repository for results from running the evaluations
#662 opened Jul 6, 2023 by c1505
7
toxigen task measures toxicity classification rather than whether generations are toxic?
#974 opened Nov 8, 2023 by laphang
7
HuggingFace model prompt formatting feature request A feature that isn't implemented yet.
#1209 opened Dec 24, 2023 by daniel-furman v0.4.3
@daniel-furman
7
Low results on TriviaQA
#1292 opened Jan 16, 2024 by yafuly
7
How to compute the perplexity only on the answer? asking questions For asking for clarification / support on library usage.
#1370 opened Jan 30, 2024 by Luobots
7
wikitext weird results Mistral-7B-v0.1 length=4096 // Gemma-7B bos missing bug Something isn't working.
#1471 opened Feb 26, 2024 by vince62s
7
Issue when Evaluating ChatGLM
#705 opened Jul 26, 2023 by Tardissss
8
Quac Dataset feature request A feature that isn't implemented yet. good first issue Good for newcomers
#827 opened Sep 4, 2023 by RanchiZhao
8
assert len(continuation_enc) error in _loglikelihood_tokens for certain (but not all) tasks?
#1053 opened Dec 2, 2023 by lhl
@haileyschoelkopf
8
Process hangs when using tensor_parallel_size and data_parallel_size together bug Something isn't working.
#1734 opened Apr 22, 2024 by harshakokel
8
Implement the SuperGLUE evaluation feature request A feature that isn't implemented yet.
#22 opened Sep 16, 2020 by StellaAthena
1 of 2 tasks
5
@zphang @thefazzer
9
Improvements to MGSM
#1391 opened Feb 2, 2024 by leocnj
9
Acc vs acc_norm asking questions For asking for clarification / support on library usage.
#1396 opened Feb 5, 2024 by sqrkl
9
TGI support - API evaluation of HF models feature request A feature that isn't implemented yet. help wanted Contributors and extra help welcome.
#869 opened Sep 19, 2023 by ManuelFay
10
NAN value for truthfulqa_mc2 on full finetuned model TinyLlama
#1340 opened Jan 23, 2024 by hahmad2008
10
Clarification on API Endpoint: /v1/completions vs /v1/chat/completions
#1637 opened Mar 26, 2024 by gerayking
@haileyschoelkopf
10
The tokenizer add_special_tokens parameter for t5 model lambada task
#1017 opened Nov 22, 2023 by daisyden
11
Inverse Scaling Tasks? feature request A feature that isn't implemented yet. good first issue Good for newcomers help wanted Contributors and extra help welcome.
#1442 opened Feb 18, 2024 by RylanSchaeffer
1
@h-albert-lee
11
HellaSwag with UnicodeDecodeError
#1757 opened Apr 27, 2024 by Hua-rookie
11
Wandb logger can't handle groups with heterogenous metrics
#1958 opened Jun 12, 2024 by dmitrii-palisaderesearch
11
When using parallelize=True, raise Runtime Error: expected all tensors to be on the same device bug Something isn't working.
#1575 opened Mar 14, 2024 by feiba54
@haileyschoelkopf
13
ProTip! Mix and match filters to narrow down what you’re looking for.