-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Issues: EleutherAI/lm-evaluation-harness
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
Add a docs FAQ section
documentation
Improvements or additions to documentation.
#1676
opened Apr 5, 2024 by
haileyschoelkopf
Support for sequence tagging tasks
asking questions
For asking for clarification / support on library usage.
#1675
opened Apr 5, 2024 by
Khalid-Nabigh
log_samples with multi args in model_args
bug
Something isn't working.
#1664
opened Apr 3, 2024 by
nicho2
Evaluating multiple choice questions with GPT (OpenAI Chat Completion API)
#1662
opened Apr 3, 2024 by
kangqi-ni
ERROR: file or directory not found: /data/liuhuanbin/code/assessments/lm-evaluation-harness/tests/test_version_stable.py
bug
Something isn't working.
#1658
opened Apr 2, 2024 by
LHB-kk
understanding context length behaviors
asking questions
For asking for clarification / support on library usage.
#1642
opened Mar 27, 2024 by
simran-arora
Clarification on API Endpoint: /v1/completions vs /v1/chat/completions
#1637
opened Mar 26, 2024 by
gerayking
How can I evaluate on an output file?
feature request
A feature that isn't implemented yet.
#1627
opened Mar 24, 2024 by
Luobots
Add alternate (configurable) launcher / orchestration + sweep functionality
#1622
opened Mar 22, 2024 by
haileyschoelkopf
Allow registering custom LM implementations without requiring loading and modifying lm_eval code
bug
Something isn't working.
#1621
opened Mar 22, 2024 by
apetrov-msk
Add better test coverage for models
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
help wanted
Contributors and extra help welcome.
#1613
opened Mar 20, 2024 by
haileyschoelkopf
Make managing task variants / subversions easier
feature request
A feature that isn't implemented yet.
#1602
opened Mar 18, 2024 by
haileyschoelkopf
Negative perplexity values
asking questions
For asking for clarification / support on library usage.
#1595
opened Mar 17, 2024 by
shikhar-srivastava
Make Adding New MCQA Metrics Easier
feature request
A feature that isn't implemented yet.
#1585
opened Mar 15, 2024 by
haileyschoelkopf
Sanity checking the semantic meaning of "perplexity" in code
asking questions
For asking for clarification / support on library usage.
#1581
opened Mar 15, 2024 by
RylanSchaeffer
(Question) How can I fully utilize the number of cores in my CPU?
#1576
opened Mar 14, 2024 by
WCSY-YG
When using Something isn't working.
parallelize=True
, raise Runtime Error: expected all tensors to be on the same device
bug
#1575
opened Mar 14, 2024 by
feiba54
Expose Configuration Options for Perplexity calculations
feature request
A feature that isn't implemented yet.
#1565
opened Mar 12, 2024 by
haileyschoelkopf
Allow Task objects to defer dataset download
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
help wanted
Contributors and extra help welcome.
#1558
opened Mar 11, 2024 by
haileyschoelkopf
Whitespace before label in MultipleChoiceTask causes wrong label probability prediction
#1556
opened Mar 11, 2024 by
RibinMTC
ProTip!
Updated in the last three days: updated:>2024-07-15.