-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Issues: EleutherAI/lm-evaluation-harness
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
Add New Lambada Translations
good first issue
Good for newcomers
#1501
opened Feb 29, 2024 by
haileyschoelkopf
Make managing task variants / subversions easier
feature request
A feature that isn't implemented yet.
#1602
opened Mar 18, 2024 by
haileyschoelkopf
Negative perplexity values
asking questions
For asking for clarification / support on library usage.
#1595
opened Mar 17, 2024 by
shikhar-srivastava
Make Adding New MCQA Metrics Easier
feature request
A feature that isn't implemented yet.
#1585
opened Mar 15, 2024 by
haileyschoelkopf
(Question) How can I fully utilize the number of cores in my CPU?
#1576
opened Mar 14, 2024 by
WCSY-YG
Expose Configuration Options for Perplexity calculations
feature request
A feature that isn't implemented yet.
#1565
opened Mar 12, 2024 by
haileyschoelkopf
New Task Request: InflectionAI's Physics GRE
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
help wanted
Contributors and extra help welcome.
#1554
opened Mar 10, 2024 by
RylanSchaeffer
Run pawsx task got "TypeError: 'NoneType' object cannot be interpreted as an integer" error.
bug
Something isn't working.
#1539
opened Mar 7, 2024 by
weizhixiaoyi
concurrent api request to accelerate evaluation
feature request
A feature that isn't implemented yet.
#1504
opened Mar 1, 2024 by
jordane95
how to add tasks with requests based on the answers for the previous requests?
#1432
opened Feb 16, 2024 by
artemorloff
Output format of samples has been changed
bug
Something isn't working.
#1493
opened Feb 28, 2024 by
christyler3030
Issue with
bigbench_gender_inclusive_sentences_german_multiple_choice
#1473
opened Feb 26, 2024 by
ayulockin
evaluation extremely slow with llama_cpp/gguf
bug
Something isn't working.
#1472
opened Feb 26, 2024 by
mobicham
wikitext weird results Mistral-7B-v0.1 length=4096 // Gemma-7B bos missing
bug
Something isn't working.
#1471
opened Feb 26, 2024 by
vince62s
log_samples File name too long. Need truncation or override
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
help wanted
Contributors and extra help welcome.
#1454
opened Feb 21, 2024 by
ryxli
janitor_util C++ splits multibyte characters into non-UTF bytes(?)
#1452
opened Feb 21, 2024 by
mycoalchen
ProTip!
Exclude everything labeled
bug
with -label:bug.