docs: Clarify that custom code evals are not accepted #963
Conversation
README.md
Outdated
| @@ -13,6 +13,9 @@ To get set up with evals, follow the [setup instructions below](README.md#Setup) | |||
| - Familiarize yourself with the existing eval templates: [eval-templates.md](docs/eval-templates.md). | |||
|
|
|||
| #### Writing evals | |||
|
|
|||
| **Important: Please note that we are currenlty not accepting Evals with custom code, with the exception of modelgraded evals with custom code.** | |||
There was a problem hiding this comment.
We are accepting modelgraded evals with custom modelgraded YAML files, for example they contribute a new .yaml file to the registry here: https://github.com/openai/evals/tree/main/evals/registry/modelgraded
There was a problem hiding this comment.
So to clarify, we will accept PRs that have new .yaml and new .jsonl (in git-lfs) files, but if they wrote new .py files, we unfortunately can't review them right now
There was a problem hiding this comment.
Okay, I will adapt the PR accordingly.
There was a problem hiding this comment.
Please re-review @andrew-openai.
|
@andrew-openai would you be able to confirm that having custom .js files which help with building sample.jsonl inputs are okay? #981 I followed an example of an eval that was merged previously which followed this same pattern but can remove the javascript files if they are not valuable/allowed. |
## Description This PR provides a clear info statement, reg. the fact that custom code evals are currently not accepted.
## Description This PR provides a clear info statement, reg. the fact that custom code evals are currently not accepted.
Description
This PR provides a clear info statement, reg. the fact that custom code evals are currently not accepted.