Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation about tasks <> request types mapping #1202

Open
anjor opened this issue Dec 22, 2023 · 1 comment
Open

Documentation about tasks <> request types mapping #1202

anjor opened this issue Dec 22, 2023 · 1 comment

Comments

@anjor
Copy link
Contributor

anjor commented Dec 22, 2023

This table in the readme outlining which request type is supported for each API is super useful.
Screenshot 2023-12-22 at 11 06 07

However, as a user I usually want to run a set of benchmarks (tasks) against a set of APIs. Not knowing how the tasks map to request types means I don't know until I actually try to run the benchmark whether or not the run is going to succeed.

For example, running hellaswag with openai chat completions fails because loglikelihood and loglikelihood_rolling are not implemented due to lack of support for logits.

@haileyschoelkopf
Copy link
Contributor

Will complete as part of #1160 ! This is very useful feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants