-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Issues: EleutherAI/lm-evaluation-harness
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Author
Label
Projects
Milestones
Assignee
Sort
Issues list
Integrate Semantic Answer Similarity (SAS) into the evaluation metrics.
#1703
opened Apr 15, 2024 by
gonzalo-santamaria-iic
Module 'exact_match' doesn't exist on the Hugging Face Hub either.
#1697
opened Apr 12, 2024 by
BBaekdabang
[TruthfulQA] update rouge-score version or add a way to suppress tokenizer logging
#1692
opened Apr 9, 2024 by
skramer-dev
Better Document Data-Parallel interface / clean it up
feature request
A feature that isn't implemented yet.
#1684
opened Apr 7, 2024 by
haileyschoelkopf
Cleanup Dependencies Further
feature request
A feature that isn't implemented yet.
#1683
opened Apr 7, 2024 by
haileyschoelkopf
Add docstring for HFLM's many keyword args
documentation
Improvements or additions to documentation.
feature request
A feature that isn't implemented yet.
good first issue
Good for newcomers
help wanted
Contributors and extra help welcome.
#1682
opened Apr 7, 2024 by
haileyschoelkopf
Add Improvements or additions to documentation.
nemo
LM class to table of supported models / libraries
documentation
#1681
opened Apr 7, 2024 by
haileyschoelkopf
List of A feature that isn't implemented yet.
until
tokens in cli args not supported
feature request
#1677
opened Apr 5, 2024 by
tianyi-chen
Add a docs FAQ section
documentation
Improvements or additions to documentation.
#1676
opened Apr 5, 2024 by
haileyschoelkopf
Support for sequence tagging tasks
asking questions
For asking for clarification / support on library usage.
#1675
opened Apr 5, 2024 by
Khalid-Nabigh
log_samples with multi args in model_args
bug
Something isn't working.
#1664
opened Apr 3, 2024 by
nicho2
Evaluating multiple choice questions with GPT (OpenAI Chat Completion API)
#1662
opened Apr 3, 2024 by
kangqi-ni
ERROR: file or directory not found: /data/liuhuanbin/code/assessments/lm-evaluation-harness/tests/test_version_stable.py
bug
Something isn't working.
#1658
opened Apr 2, 2024 by
LHB-kk
understanding context length behaviors
asking questions
For asking for clarification / support on library usage.
#1642
opened Mar 27, 2024 by
simran-arora
Clarification on API Endpoint: /v1/completions vs /v1/chat/completions
#1637
opened Mar 26, 2024 by
gerayking
ProTip!
Updated in the last three days: updated:>2024-06-20.