-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Issues: EleutherAI/lm-evaluation-harness
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
Evaluating multiple choice questions with GPT (OpenAI Chat Completion API)
#1662
opened Apr 3, 2024 by
kangqi-ni
Better Document Data-Parallel interface / clean it up
feature request
A feature that isn't implemented yet.
#1684
opened Apr 7, 2024 by
haileyschoelkopf
Cleanup Dependencies Further
feature request
A feature that isn't implemented yet.
#1683
opened Apr 7, 2024 by
haileyschoelkopf
Add docstring for HFLM's many keyword args
documentation
Improvements or additions to documentation.
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
help wanted
Contributors and extra help welcome.
#1682
opened Apr 7, 2024 by
haileyschoelkopf
Add Improvements or additions to documentation.
nemo
LM class to table of supported models / libraries
documentation
#1681
opened Apr 7, 2024 by
haileyschoelkopf
List of A feature that isn't implemented yet.
until
tokens in cli args not supported
feature request
#1677
opened Apr 5, 2024 by
tianyi-chen
Add a docs FAQ section
documentation
Improvements or additions to documentation.
#1676
opened Apr 5, 2024 by
haileyschoelkopf
Support for sequence tagging tasks
asking questions
For asking for clarification / support on library usage.
#1675
opened Apr 5, 2024 by
Khalid-Nabigh
log_samples with multi args in model_args
bug
Something isn't working.
#1664
opened Apr 3, 2024 by
nicho2
Negative perplexity values
asking questions
For asking for clarification / support on library usage.
#1595
opened Mar 17, 2024 by
shikhar-srivastava
ERROR: file or directory not found: /data/liuhuanbin/code/assessments/lm-evaluation-harness/tests/test_version_stable.py
bug
Something isn't working.
#1658
opened Apr 2, 2024 by
LHB-kk
understanding context length behaviors
asking questions
For asking for clarification / support on library usage.
#1642
opened Mar 27, 2024 by
simran-arora
Clarification on API Endpoint: /v1/completions vs /v1/chat/completions
#1637
opened Mar 26, 2024 by
gerayking
How can I evaluate on an output file?
feature request
A feature that isn't implemented yet.
#1627
opened Mar 24, 2024 by
Luobots
Add alternate (configurable) launcher / orchestration + sweep functionality
#1622
opened Mar 22, 2024 by
haileyschoelkopf
Allow registering custom LM implementations without requiring loading and modifying lm_eval code
bug
Something isn't working.
#1621
opened Mar 22, 2024 by
apetrov-msk
Add better test coverage for models
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
help wanted
Contributors and extra help welcome.
#1613
opened Mar 20, 2024 by
haileyschoelkopf
Make managing task variants / subversions easier
feature request
A feature that isn't implemented yet.
#1602
opened Mar 18, 2024 by
haileyschoelkopf
ProTip!
Updated in the last three days: updated:>2024-07-16.