You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This table in the readme outlining which request type is supported for each API is super useful.
However, as a user I usually want to run a set of benchmarks (tasks) against a set of APIs. Not knowing how the tasks map to request types means I don't know until I actually try to run the benchmark whether or not the run is going to succeed.
For example, running hellaswag with openai chat completions fails because loglikelihood and loglikelihood_rolling are not implemented due to lack of support for logits.
The text was updated successfully, but these errors were encountered:
This table in the readme outlining which request type is supported for each API is super useful.
![Screenshot 2023-12-22 at 11 06 07](https://private-user-images.githubusercontent.com/1911631/292472887-7d4b4b39-633f-4f41-be8c-264543edfe59.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTg4ODExMjMsIm5iZiI6MTcxODg4MDgyMywicGF0aCI6Ii8xOTExNjMxLzI5MjQ3Mjg4Ny03ZDRiNGIzOS02MzNmLTRmNDEtYmU4Yy0yNjQ1NDNlZGZlNTkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyMCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjBUMTA1MzQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZTlmODk4MTBlYmQyYTFkNDliZTM4NGRmYjk2MTYyYjQxZDUyYTExMmRlMzgwODBlZGFiMzlhMDVjZTUwNmE5YiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.nEsMmlV0Ht2nrtCD2hHA1e37gfT-hMYOGNxakC6SQBY)
However, as a user I usually want to run a set of benchmarks (tasks) against a set of APIs. Not knowing how the tasks map to request types means I don't know until I actually try to run the benchmark whether or not the run is going to succeed.
For example, running hellaswag with openai chat completions fails because
loglikelihood
andloglikelihood_rolling
are not implemented due to lack of support for logits.The text was updated successfully, but these errors were encountered: