Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clear current search query, filters, and sorts
282 Open 807 Closed
282 Open 807 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

fewshot for datasets where there's only one set
#165 by leogao2 was closed Nov 1, 2023
fix CoLA bug Something isn't working.
#171 by leogao2 was closed Nov 21, 2022
testing: log task docs, requests, results etc and ensure no change
#174 by leogao2 was closed Nov 1, 2023
testing: ensure distance from gpt3 paper numbers not too far
#175 by leogao2 was closed Mar 9, 2023
1 task
Implement the TweetQA evaluation feature request A feature that isn't implemented yet.
#185 by leogao2 was closed Nov 21, 2022
add versioning
#186 by leogao2 was closed Jun 5, 2021
table generator
#187 by leogao2 was closed Mar 9, 2023
@StellaAthena
get docs set up
#188 by leogao2 was closed Mar 9, 2023
versioning test
#189 by leogao2 was closed Jun 12, 2021
Add XNLI feature request A feature that isn't implemented yet.
#199 by sdtblck was closed Mar 15, 2024
Add MKQA feature request A feature that isn't implemented yet.
#208 by sdtblck was closed Nov 21, 2022
remove annoying syntax sugar behavior from loglikelihood
#218 by leogao2 was closed Mar 9, 2023
fix descriptions thing
#220 by leogao2 was closed Feb 13, 2022
Implement the Contract Understanding Atticus Dataset (CUAD) feature request A feature that isn't implemented yet.
#221 by StellaAthena was closed Nov 21, 2022
Implement SCAN feature request A feature that isn't implemented yet.
#224 by StellaAthena was closed Nov 21, 2022
BUG: TypeError: create_from_arg_string() takes 2 positional arguments but 3 were given
#255 by mrseeker was closed Feb 12, 2022
1
Add extractive spans request type to RequestFactory
#284 by StephenHogg was closed Mar 25, 2023
Add extractive spans to QASPER
#285 by StephenHogg was closed Mar 25, 2023
Add textsynth.com API support
#295 by bellard was closed Nov 21, 2022
Add language argument
#298 by hyunwoongko was closed Nov 8, 2023
Add SWAG ( https://rowanzellers.com/swag/ )
#304 by BlinkDL was closed Apr 28, 2022
1
ModuleNotFoundError: No module named 'janitor_util'
#323 by Muennighoff was closed May 11, 2022
TypeError: expected str, bytes or os.PathLike object, not NoneType
#342 by kabbi159 was closed Sep 16, 2022
How can I use this evaluation in a cluster blocked for hugging face server?
#353 by a-cavalcanti was closed Nov 13, 2022
Is there some problem with the evaluation code?
#354 by a-cavalcanti was closed Nov 7, 2022
ProTip! Find all open issues with in progress development work with linked:pr.