Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Verify Stopsequences Don't Impact Scores #1086

Open
haileyschoelkopf opened this issue Dec 9, 2023 · 0 comments
Open

Verify Stopsequences Don't Impact Scores #1086

haileyschoelkopf opened this issue Dec 9, 2023 · 0 comments
Labels
validation For validation of task implementations.

Comments

@haileyschoelkopf
Copy link
Contributor

We should check on all tasks to ensure that they do not use stop sequences that might cut off valid answers too early, such as happened with DROP. We've already done so for some tasks such as GSM8k as of the new release, but should look at other generation-based tasks.

A notable culprit is using \n as a stop sequence when \n\n would be more correct.
@haileyschoelkopf haileyschoelkopf added the validation For validation of task implementations. label Dec 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
validation For validation of task implementations.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@haileyschoelkopf