-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Issues: EleutherAI/lm-evaluation-harness
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
Negative perplexity values
asking questions
For asking for clarification / support on library usage.
#1595
opened Mar 17, 2024 by
shikhar-srivastava
Make Adding New MCQA Metrics Easier
feature request
A feature that isn't implemented yet.
#1585
opened Mar 15, 2024 by
haileyschoelkopf
Sanity checking the semantic meaning of "perplexity" in code
asking questions
For asking for clarification / support on library usage.
#1581
opened Mar 15, 2024 by
RylanSchaeffer
(Question) How can I fully utilize the number of cores in my CPU?
#1576
opened Mar 14, 2024 by
WCSY-YG
When using Something isn't working.
parallelize=True
, raise Runtime Error: expected all tensors to be on the same device
bug
#1575
opened Mar 14, 2024 by
feiba54
Expose Configuration Options for Perplexity calculations
feature request
A feature that isn't implemented yet.
#1565
opened Mar 12, 2024 by
haileyschoelkopf
Allow Task objects to defer dataset download
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
help wanted
Contributors and extra help welcome.
#1558
opened Mar 11, 2024 by
haileyschoelkopf
Whitespace before label in MultipleChoiceTask causes wrong label probability prediction
#1556
opened Mar 11, 2024 by
RibinMTC
New Task Request: InflectionAI's Physics GRE
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
help wanted
Contributors and extra help welcome.
#1554
opened Mar 10, 2024 by
RylanSchaeffer
Run pawsx task got "TypeError: 'NoneType' object cannot be interpreted as an integer" error.
bug
Something isn't working.
#1539
opened Mar 7, 2024 by
weizhixiaoyi
evaluation extremely slow with llama_cpp/gguf
bug
Something isn't working.
#1472
opened Feb 26, 2024 by
mobicham
ValueError: Tasks not found: persona_desire-for-acquiring-eval-results.
#1512
opened Mar 3, 2024 by
RylanSchaeffer
concurrent api request to accelerate evaluation
feature request
A feature that isn't implemented yet.
#1504
opened Mar 1, 2024 by
jordane95
Add New Lambada Translations
good first issue
Good for newcomers
#1501
opened Feb 29, 2024 by
haileyschoelkopf
Output format of samples has been changed
bug
Something isn't working.
#1493
opened Feb 28, 2024 by
christyler3030
Proper way to add arguments to chosen metrics?
asking questions
For asking for clarification / support on library usage.
#1483
opened Feb 27, 2024 by
LSinev
Issue with
bigbench_gender_inclusive_sentences_german_multiple_choice
#1473
opened Feb 26, 2024 by
ayulockin
ProTip!
Updated in the last three days: updated:>2024-06-17.