Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
44 Open 50 Closed
44 Open 50 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Add HuggingFace Text-Generation-Interface Support
#2001 opened Jun 19, 2024 by taoari
Incorrect Multilingual arc implementation
#2000 opened Jun 19, 2024 by hynky1999
Implementing lessons from OLMES
#2002 opened Jun 20, 2024 by lintangsutawika
TypeError of scrolls_narrativeqa
#1891 opened May 27, 2024 by hicleo
@lintangsutawika
Duplicate sample entries
#2025 opened Jun 26, 2024 by baberabb
Running evaluation on Gemma-2 27B model
#2063 opened Jul 4, 2024 by zeynepgulhanuslu
The response is too short to extract answer on GPQA. What should I set to extend it?
#2081 opened Jul 8, 2024 by URRealHero
max_new_tokens and max_length conflict
#2070 opened Jul 5, 2024 by meg-huggingface
@haileyschoelkopf
How to RUN BENCHMARK on GGUF Models ?
#2086 opened Jul 10, 2024 by RakshitAralimatti
filter <class 'utils.WordSortFilter'> is not registered!
#2108 opened Jul 16, 2024 by Mr-Nobody1
A simple tutorial using python and not the CLI
#2117 opened Jul 19, 2024 by nprime496
lm_eval --model hf --model_args pretrained=/home/lmz/glm-4-9b-chat --tasks /home/lmz/ceval-exam/a.yaml --device cuda:0 --batch_size 8 In this command, can the --tasks option only download the dataset from the hub? Can I use a local dataset?
#2120 opened Jul 20, 2024 by LeeGitHub1
Add more math evaluation tasks
#1869 opened May 22, 2024 by jordane95
1
What is the output_type in the metric for?
#1976 opened Jun 17, 2024 by dennisrall
1
Error when chat template is not a string
#1962 opened Jun 13, 2024 by djstrong
1
Question: Realtoxicityprompts takes >10 seconds per query, is this expected behavior?
#2096 opened Jul 12, 2024 by meg-huggingface
1
The evaluation results are inconsistent across different GPUs
#2077 opened Jul 8, 2024 by DonteFlynn
1
Empty --log_samples outputs
#2115 opened Jul 19, 2024 by IsraelAbebe
1
ValueError when task name collide with local directory names
#2122 opened Jul 20, 2024 by alat-rights
@haileyschoelkopf
1
Wrong calculation of score when there are ties?
#2007 opened Jun 21, 2024 by apohllo
2
Ubelievable long time when host the gguf mode ?
#1971 opened Jun 16, 2024 by hzgdeerHo
2
Long time testing Qwen2-72B bug Something isn't working.
#1984 opened Jun 18, 2024 by djstrong
2
coqa not working bug Something isn't working.
#1529 opened Mar 5, 2024 by lchu-ibm
@haileyschoelkopf
2
Evaluate Gemma with Chat Template
#2069 opened Jul 5, 2024 by pyf98
1
2
Invalid modification to task YAML file
#2076 opened Jul 8, 2024 by duanhx1037
2
ProTip! Follow long discussions with comments:>50.