Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
210 Open 702 Closed
210 Open 702 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Per-sample perplexity of a continuation?
#2040 by YilunZhou was closed Jun 29, 2024
2
The problem of generate responses with my own trained model
#2035 by marvelcell was closed Jun 29, 2024
1
Using chat template with vllm engine
#2033 by mohit-rag was closed Jun 28, 2024
1
vllm backend faild
#2028 by chunniunai220ml was closed Jun 27, 2024
4
YAML config was updated, but the project still remains the same as before
#2021 by 2018211801 was closed Jun 27, 2024
3
Does it support Triton server？ asking questions For asking for clarification / support on library usage.
#2018 by AndyZZt was closed Jun 25, 2024
1
Running on custom model, getting 'TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'
#2016 by Fchaubard was closed Jun 24, 2024
mmlu evaluation fail
#2005 by jxiw was closed Jun 21, 2024
2
piqa task need add trust_remote_code true in piqa.yml
#1985 by changwangss was closed Jun 19, 2024
How to enable trust_remote_code when encountered programmatically via get_task_dict?
#1980 by Jack-Khuu was closed Jun 18, 2024
3
Add a way to instantiate from HF.AutoModel (again)
#1978 by dmitrii-palisaderesearch was closed Jun 19, 2024
2
incomplete task list
#1972 by hlzhang109 was closed Jun 19, 2024
2
TemplateLM#_encode_pair() only works for HF transformers auto-models
#1966 by Birch-san was closed Jun 14, 2024
1
Error while installing
#1965 by surya-narayanan was closed Jun 19, 2024
1
Multi-gpu evaluation with external library usage.
#1960 by xinghaow99 was closed Jun 13, 2024
1
Cannot load model 'local-chat-completions' and 'local-completions'
#1957 by awesom112 was closed Jun 12, 2024
Keep getting error: 'VLLM' object has no attribute 'AUTO_MODEL_CLASS'
#1953 by andrew0411 was closed Jun 12, 2024
7
.ipynb_checkpoints causes eval harness to fail
#1952 by johnwee1 was closed Jun 13, 2024
1
Results is weird for Qwen2-1.5B
#1944 by SefaZeng was closed Jun 24, 2024
6
Save fewshot_as_multiturn argument in results.json
#1941 by djstrong was closed Jun 19, 2024
1
1
Format of Personal Defined Dataset for Evaluation
#1937 by OscarC9912 was closed Jun 9, 2024
1
Parallel GPU evaluation using simple_evaluate /evaluate functions? #1934
#1935 by PalaashAgrawal was closed Jun 7, 2024
1
Parallel GPU evaluation using simple_evaluate /evaluate functions?
#1934 by Naitik1502 was closed Jun 7, 2024
--trust_remote_code does it actually do anything? bug Something isn't working.
#1932 by devzzzero was closed Jun 19, 2024
1
@haileyschoelkopf
8
build commit_id=b281b09, I cannot find lm-eval command.
#1920 by jieheroli was closed Jun 4, 2024
1
ProTip! Mix and match filters to narrow down what you’re looking for.