Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add task variants replicating Llama 1 / 2 evaluation numbers #1078

Open
haileyschoelkopf opened this issue Dec 7, 2023 · 1 comment
Open
Assignees
Labels
feature request A feature that isn't implemented yet.

Comments

@haileyschoelkopf
Copy link
Collaborator

In some cases, Llama 1 (and 2, whose eval setups at times differ) paper results are not replicable by our implementations due to Meta’s custom undisclosed prompts or prepended task descriptions.

However, for some tasks like Triviaqa, we have successfully found setups / reverse engineered prompts. Where we have done this we should add documentation and variants of the tasks for ease of use.

@haileyschoelkopf haileyschoelkopf added the feature request A feature that isn't implemented yet. label Dec 7, 2023
@haileyschoelkopf haileyschoelkopf self-assigned this Dec 7, 2023
@afcruzs
Copy link

afcruzs commented Mar 16, 2024

@haileyschoelkopf Is this documented yet somewhere? Even if it's in an informal capacity (eg a branch, a Google doc etc)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request A feature that isn't implemented yet.
Projects
None yet
Development

No branches or pull requests

2 participants