Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose Configuration Options for Perplexity calculations #1565

Open
haileyschoelkopf opened this issue Mar 12, 2024 · 0 comments
Open

Expose Configuration Options for Perplexity calculations #1565

haileyschoelkopf opened this issue Mar 12, 2024 · 0 comments
Labels
feature request A feature that isn't implemented yet.

Comments

@haileyschoelkopf
Copy link
Contributor

At some point we should add options for PPL calculations, such as the ability to determine whether the windows overlap with each other, and .

Note that this heavily impacts the speed / compute required for evaluations in addition to scores.

Preferred implementation approach:

  • convert the generation_kwargs TaskConfig field to request_kwargs, leaving back-compatibility for providing generation_kwargs if request_kwargs is not in a given config
  • allow loglikelihood_rolling requests to take these request_kwargs, and implement new helper functions different from the current creation of context windows for loglikelihood rolling, and add unit tests for these helper fns
  • add some example task using these options
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request A feature that isn't implemented yet.
Projects
Status: Backlog
Development

No branches or pull requests

1 participant