[evals] Refactor evals package to expose `completion_fn`. #515

hwchung27 · 2023-03-30T00:54:41Z

PAIR=jasonwei

Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, failure to follow the guidelines below will result in the PR being closed automatically. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨

PLEASE READ THIS:

In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task.

We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples.

Eval details 📑

Eval name

[Insert Eval name here]

Eval description

[Insert a short description of what your eval does here]

What makes this a useful eval?

[Insert why this eval is worth including and any additional context]

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100)

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

Insert what makes your eval high quality that was not mentioned above. (Not required)

Eval structure 🏗️

Your eval should

Check that your data is in evals/registry/data/{name}
Check that your yaml is registered at evals/registry/evals/{name}.yaml
Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.

I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgement

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.

Submit eval

I have filled out all required fields in the evals PR form
(Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

INSERT_EVAL_HERE

PAIR=jasonwei Co-authored-by: Jason Wei <[email protected]>

andrew-openai

LGTM, I love CompletionFn. Let's try to refactor the codebase so that it's more functional and let's you pass around instances of CompletionFn (see other comment in review)

andrew-openai · 2023-03-31T23:26:16Z

evals/api.py

@@ -97,6 +97,7 @@ def completion_query(
 return result, openai_create_prompt, metadata


+# TODO(hwc): remove this


I believe we concluded we'll keep it but refactor it to use new fns?

andrew-openai · 2023-03-31T23:28:58Z

evals/api.py

- expected = [expected]
- if options is None:
- options = expected
-
 result, actual_prompt, metadata = completion_query(


Do we need to rewrite completion_query with new fn?

andrew-openai · 2023-03-31T23:52:34Z

evals/elsuite/utils.py

@@ -135,3 +144,31 @@ def __call__(self, **kwargs):
 **self.completion_kwargs,
 )
 return completion, prompt
+
+
+class CompletionFn(Protocol):


In general I like CompletionFn but there is some organization to be done, for example:

When to use CompletionFn versus completion_query

Should some of the utility functionality like sample_freeform use CompletionFn

Should we separate implementations of CompletionFn from recording/evaluating utilities

Should completion_query be a subclass of CompletionFn (and be renamed to something like openai_completion_query)

Let's discuss?

Ideally I'd like to have evals only use CompletionFn as opposed to picking between CompletionFn and completion_query (more accurately openai_completion_query as @andrew-openai pointed out).

Also happy to discuss if needed.

One more thing to consider: We need CompletionFn subclasses to probably support both chat and non-chat inputs, which means implementing some generic casting behavior to go from chat to non-chat. I think luckily we have a lot of this already, which is implemented in PromptFn and chat_prompt_to_text_prompt, but just need to add it to CompletionFn

The refactor needs to extend through the codebase, something like:

. ├── completion_fns │ ├── __init__.py │ ├── completion_fn.py (contains CompletionFn protocol) │ ├── openai_completion_fn.py (contains OpenAICompletionFn implementation) │ └── ... (other implementations) ├── api.py (updated to use CompletionFn instances) ├── utils.py (updated to use the refactored api.py and CompletionFn implementations) └── ... (other existing modules)

Here is my current proposal for making api.py a bit neater:

Rewrite sample_freeform to support arbitrary CompletionFns.

Rewrite check_sampled_text to use sample_freeform and record_and_check_match. Remove record_sampling call from record_and_check_match. check_sampled_text to be renamed to check_match_sampled_text and moved to elsuite.utils.

completion_query as a subclass of CompletionFn

CompletionFn to be moved to api`.

evals.Eval has required argument completion_fn

evals/elsuite/utils.py

jwang47 · 2023-04-03T18:37:00Z

evals/elsuite/utils.py

@@ -135,3 +144,31 @@ def __call__(self, **kwargs):
 **self.completion_kwargs,
 )
 return completion, prompt
+
+
+class CompletionFn(Protocol):


Ideally I'd like to have evals only use CompletionFn as opposed to picking between CompletionFn and completion_query (more accurately openai_completion_query as @andrew-openai pointed out).

Also happy to discuss if needed.

evals/eval.py

evals/api.py

evals/elsuite/basic/includes.py

# Thank you for contributing an eval! ♥️ 🚨 Please make sure your PR follows these guidelines, __failure to follow the guidelines below will result in the PR being closed automatically__. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨 __PLEASE READ THIS__: In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task. We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples. ## Eval details 📑 ### Eval name [Insert Eval name here] ### Eval description [Insert a short description of what your eval does here] ### What makes this a useful eval? [Insert why this eval is worth including and any additional context] ## Criteria for a good eval ✅ Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals). Your eval should be: - [ ] Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world. - [ ] Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not. - [ ] Includes good signal around what is the right behavior. This means either a correct answer for `Basic` evals or the `Fact` Model-graded eval, or an exhaustive rubric for evaluating answers for the `Criteria` Model-graded eval. - [ ] Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100) If there is anything else that makes your eval worth including, please document it below. ### Unique eval value > Insert what makes your eval high quality that was not mentioned above. (Not required) ## Eval structure 🏗️ Your eval should - [ ] Check that your data is in `evals/registry/data/{name}` - [ ] Check that your yaml is registered at `evals/registry/evals/{name}.yaml` - [ ] Ensure you have the right to use the data you submit via this eval (For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.) ## Final checklist 👀 ### Submission agreement By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies). - [ ] I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies. ### Email address validation If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request. - [ ] I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request. ### Limited availability acknowledgement We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR. - [ ] I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted. ### Submit eval - [ ] I have filled out all required fields in the evals PR form - [ ] (Ignore if not submitting code) I have run `pip install pre-commit; pre-commit install` and have verified that `black`, `isort`, and `autoflake` are running when I commit and push Failure to fill out all required fields will result in the PR being closed. ### Eval JSON data Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here: <details> <summary>View evals in JSON</summary> ### Eval ```jsonl INSERT_EVAL_HERE ``` </details>

…#592) This supports downloading all data on pip install from git, e.g. `pip install git+https://github.com/openai/evals.git@main`

…ed evals

# Thank you for contributing an eval! ♥️ 🚨 Please make sure your PR follows these guidelines, __failure to follow the guidelines below will result in the PR being closed automatically__. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨 __PLEASE READ THIS__: In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task. We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples. ## Eval details 📑 ### Eval name [Insert Eval name here] ### Eval description [Insert a short description of what your eval does here] ### What makes this a useful eval? [Insert why this eval is worth including and any additional context] ## Criteria for a good eval ✅ Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals). Your eval should be: - [ ] Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world. - [ ] Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not. - [ ] Includes good signal around what is the right behavior. This means either a correct answer for `Basic` evals or the `Fact` Model-graded eval, or an exhaustive rubric for evaluating answers for the `Criteria` Model-graded eval. - [ ] Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100) If there is anything else that makes your eval worth including, please document it below. ### Unique eval value > Insert what makes your eval high quality that was not mentioned above. (Not required) ## Eval structure 🏗️ Your eval should - [ ] Check that your data is in `evals/registry/data/{name}` - [ ] Check that your yaml is registered at `evals/registry/evals/{name}.yaml` - [ ] Ensure you have the right to use the data you submit via this eval (For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.) ## Final checklist 👀 ### Submission agreement By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies). - [ ] I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies. ### Email address validation If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request. - [ ] I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request. ### Limited availability acknowledgement We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR. - [ ] I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted. ### Submit eval - [ ] I have filled out all required fields in the evals PR form - [ ] (Ignore if not submitting code) I have run `pip install pre-commit; pre-commit install` and have verified that `black`, `isort`, and `autoflake` are running when I commit and push Failure to fill out all required fields will result in the PR being closed. ### Eval JSON data Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here: <details> <summary>View evals in JSON</summary> ### Eval ```jsonl INSERT_EVAL_HERE ``` </details>

Replace ModelSpec with CompletionFn and allow users to specify CompletionFn instances from the CLI. Testing done: ``` oaievalset dummy test --max_samples 1 oaievalset gpt-3.5-turbo test --max_samples 1 oaievalset testing test --max_samples 1 ``` --------- Co-authored-by: Andrew Kondrich <[email protected]>

andrew-openai · 2023-04-06T23:18:44Z

evals/cli/oaieval.py

@@ -26,10 +24,8 @@ def _purple(str):

 def get_parser() -> argparse.ArgumentParser:
 parser = argparse.ArgumentParser(description="Run evals through the API")
- parser.add_argument("model", type=str, help="Name of a completion model.")
+ parser.add_argument("completion_fn", type=str, help="One or more CompletionFn URLs, separated by commas (,). The format of a CompletionFn URL can be two forms: 1) an OpenAI API model followed by query parameters (e.g. `gpt-3.5-turbo?api_key=..`) or 2) a path to a Python class followed by query parameters (e.g. `evals.api:OpenAICompletionFn?model=text-davinci-003`).")


do we need to update the help here?

- [evals] Refactor evals package to expose `completion_fn`. - Add `record_raw_samples` - Andrew/evals refactor (#579) - update manifest and pyproject to support fetching data on pip install (#592) - we need to still use the interop for string/list[dicts] for modelgraded evals - refactor simple evals to not use result.prompt (#593) - Clean up duplicate recordings - Replace ModelSpecs with CompletionFn (#594) - Add --registry_path CLI arg # Thank you for contributing an eval! ♥️ 🚨 Please make sure your PR follows these guidelines, __failure to follow the guidelines below will result in the PR being closed automatically__. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨 __PLEASE READ THIS__: In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task. We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples. ## Eval details 📑 ### Eval name [Insert Eval name here] ### Eval description [Insert a short description of what your eval does here] ### What makes this a useful eval? [Insert why this eval is worth including and any additional context] ## Criteria for a good eval ✅ Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals). Your eval should be: - [ ] Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world. - [ ] Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not. - [ ] Includes good signal around what is the right behavior. This means either a correct answer for `Basic` evals or the `Fact` Model-graded eval, or an exhaustive rubric for evaluating answers for the `Criteria` Model-graded eval. - [ ] Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100) If there is anything else that makes your eval worth including, please document it below. ### Unique eval value > Insert what makes your eval high quality that was not mentioned above. (Not required) ## Eval structure 🏗️ Your eval should - [ ] Check that your data is in `evals/registry/data/{name}` - [ ] Check that your yaml is registered at `evals/registry/evals/{name}.yaml` - [ ] Ensure you have the right to use the data you submit via this eval (For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.) ## Final checklist 👀 ### Submission agreement By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies). - [ ] I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies. ### Email address validation If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request. - [ ] I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request. ### Limited availability acknowledgement We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR. - [ ] I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted. ### Submit eval - [ ] I have filled out all required fields in the evals PR form - [ ] (Ignore if not submitting code) I have run `pip install pre-commit; pre-commit install` and have verified that `black`, `isort`, and `autoflake` are running when I commit and push Failure to fill out all required fields will result in the PR being closed. ### Eval JSON data Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here: <details> <summary>View evals in JSON</summary> ### Eval ```jsonl INSERT_EVAL_HERE ``` </details>

# Thank you for contributing an eval! ♥️ 🚨 Please make sure your PR follows these guidelines, __failure to follow the guidelines below will result in the PR being closed automatically__. Note that even if the criteria are met, that does not guarantee the PR will be merged nor GPT-4 access granted. 🚨 __PLEASE READ THIS__: In order for a PR to be merged, it must fail on GPT-4. We are aware that right now, users do not have access, so you will not be able to tell if the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep in mind as we run the eval, if GPT-4 gets higher than 90% on the eval, we will likely reject since GPT-4 is already capable of completing the task. We plan to roll out a way for users submitting evals to see the eval performance on GPT-4 soon. Stay tuned! Until then, you will not be able to see the eval performance on GPT-4. We encourage partial PR's with ~5-10 example that we can then run the evals on and share the results with you so you know how your eval does with GPT-4 before writing all 100 examples. ## Eval details 📑 ### Eval name [Insert Eval name here] ### Eval description [Insert a short description of what your eval does here] ### What makes this a useful eval? [Insert why this eval is worth including and any additional context] ## Criteria for a good eval ✅ Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals). Your eval should be: - [ ] Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world. - [ ] Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not. - [ ] Includes good signal around what is the right behavior. This means either a correct answer for `Basic` evals or the `Fact` Model-graded eval, or an exhaustive rubric for evaluating answers for the `Criteria` Model-graded eval. - [ ] Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100) If there is anything else that makes your eval worth including, please document it below. ### Unique eval value > Insert what makes your eval high quality that was not mentioned above. (Not required) ## Eval structure 🏗️ Your eval should - [ ] Check that your data is in `evals/registry/data/{name}` - [ ] Check that your yaml is registered at `evals/registry/evals/{name}.yaml` - [ ] Ensure you have the right to use the data you submit via this eval (For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.) ## Final checklist 👀 ### Submission agreement By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies). - [ ] I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies. ### Email address validation If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request. - [ ] I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request. ### Limited availability acknowledgement We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR. - [ ] I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted. ### Submit eval - [ ] I have filled out all required fields in the evals PR form - [ ] (Ignore if not submitting code) I have run `pip install pre-commit; pre-commit install` and have verified that `black`, `isort`, and `autoflake` are running when I commit and push Failure to fill out all required fields will result in the PR being closed. ### Eval JSON data Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here: <details> <summary>View evals in JSON</summary> ### Eval ```jsonl INSERT_EVAL_HERE ``` </details>

Inner monologue CoT increase 3.5 accuracy on `born-first` from 63% -> 93%

jwang47 · 2023-04-10T22:45:36Z

docs/run-evals.md


-In this example, `gpt-3.5-turbo` is the model to evaluate, and `test-match` is the eval to run. The valid model names are those which you have access to via the API. The valid eval names are specified in the YAML files under `evals/registry/evals`, and their corresponding implementations can be found in `evals/elsuite`.
+In this example, `gpt-3.5-turbo` is an OpenAI model that we dynamically instantiate as a completion function using `OpenAIChatCompletionFn(model=gpt-3.5-turbo)`. Any implementation of the `CompletionFn` protocol can be run against `oaieval`. By default, we support calling `oaieval` with any model availableon the OpenAI API or with CompletionFunctions available in [`evals/registry/completion_fns`](../evals/registry/completion_fns/). 


typo: availableon

jwang47 · 2023-04-10T22:46:32Z

evals/cli/oaieval.py

+ parser.add_argument(
+ "completion_fn",
+ type=str,
+ help="One or more CompletionFn URLs, separated by commas (,). The format of a CompletionFn URL can be two forms: 1) an OpenAI API model followed by query parameters (e.g. `gpt-3.5-turbo?api_key=..`) or 2) a path to a Python class followed by query parameters (e.g. `evals:OpenAICompletionFn?model=text-davinci-003`).",


Need to update this help message

jwang47 · 2023-04-10T22:49:53Z

evals/prompt/base.py

@@ -12,7 +12,7 @@
 ENCODER_LOCK = threading.Lock()

 # This is an approximation to the type accepted as the `prompt` field to `openai.Completion.create` calls
-OpenAICreatePrompt = Union[str, list[str], list[int], list[list[int]]]
+OpenAICreatePrompt = Union[str, list[str]]


I think we shouldn't change this since this is specifically for openai.Completion.create

jwang47 · 2023-04-10T22:53:01Z

evals/completion_fns/openai.py

+
+ def __call__(
+ self,
+ prompt: Union[OpenAICreatePrompt, Prompt],


Should update to prompt: Union[str, OpenAICreateChatPrompt]

jwang47 · 2023-04-10T22:53:08Z

evals/completion_fns/openai.py

+
+ def __call__(
+ self,
+ prompt: Union[OpenAICreateChatPrompt, Prompt],


Should update to prompt: Union[str, OpenAICreateChatPrompt]

jwang47 · 2023-04-10T22:53:50Z

evals/registry/completion_fns/tmp.yaml

@@ -0,0 +1,6 @@
+# for testing, remove before committing


Should remove this

pyproject.toml

jwang47 · 2023-04-11T01:01:08Z

MANIFEST.in

@@ -2,3 +2,4 @@ recursive-include evals *.py
 recursive-include evals *.yaml
 recursive-include evals *.sql
 recursive-include evals/registry/data *.jsonl
+recursive-include evals *.jsonl


We probably don't need this, can just keep the previous line: recursive-include evals/registry/data *.jsonl

jwang47 · 2023-04-11T01:03:47Z

evals/elsuite/modelgraded/classify.py

@@ -130,85 +71,6 @@ def __init__(
 else:
 self.eval_completion_fn = OpenAIChatCompletionFn(model="gpt-3.5-turbo")

- """import prompt and set attributes"""


eval_model in the constructor is no longer used, can remove it

PAIR=jasonwei - Move Evals functionality to use CompletionFns from ModelSpecs. --------- Co-authored-by: Jason Wei <[email protected]> Co-authored-by: Andrew Kondrich <[email protected]> Co-authored-by: Andrew Kondrich <[email protected]> Co-authored-by: Alvin Wang <[email protected]> Co-authored-by: joe-at-openai <[email protected]>

[evals] Refactor evals package to expose completion_fn.

d87a056

PAIR=jasonwei Co-authored-by: Jason Wei <[email protected]>

hwchung27 requested review from andrew-openai and jwang47 March 30, 2023 01:04

Add record_raw_samples

d9c1395

andrew-openai reviewed Apr 3, 2023

View reviewed changes

jwang47 reviewed Apr 3, 2023

View reviewed changes

andrew-openai reviewed Apr 3, 2023

View reviewed changes

evals/elsuite/basic/includes.py Outdated Show resolved Hide resolved

andrew-openai mentioned this pull request Apr 5, 2023

Performing evals on other models for comparison #528

Closed

andrew-openai and others added 5 commits April 5, 2023 14:20

update manifest and pyproject to support fetching data on pip install (…

deb29d3

…#592) This supports downloading all data on pip install from git, e.g. `pip install git+https://github.com/openai/evals.git@main`

we need to still use the interop for string/list[dicts] for modelgrad…

9b1c350

…ed evals

Clean up duplicate recordings

b691cfa

andrew-openai reviewed Apr 6, 2023

View reviewed changes

jwang47 and others added 10 commits April 6, 2023 16:52

Andrew/langchain llms (#602)

924d2d4

Update completion-fn-protocol.md

013d636

some documentation cleanup

08062bc

some documentation cleanup

3367006

some documentation cleanup

5e71a76

inner monologue example (#610)

e621b6f

Inner monologue CoT increase 3.5 accuracy on `born-first` from 63% -> 93%

Update README.md

49d17ed

Update run-evals.md

1bfba77

jwang47 reviewed Apr 10, 2023

View reviewed changes

jwang47 previously approved these changes Apr 10, 2023

View reviewed changes

cleanup

b018aff

andrew-openai dismissed jwang47’s stale review via b018aff April 10, 2023 23:13

Merge branch 'main' into evals_refactor_merge_main

5222f2c

get oaieval to run

9db703d

andrew-openai requested a review from rlbayes April 11, 2023 00:20

jwang47 previously approved these changes Apr 11, 2023

View reviewed changes

address comments

02bc2cb

andrew-openai dismissed jwang47’s stale review via 02bc2cb April 11, 2023 01:06

bump version

50114a5

jwang47 approved these changes Apr 11, 2023

View reviewed changes

andrew-openai merged commit 64fb72a into main Apr 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[evals] Refactor evals package to expose `completion_fn`. #515

[evals] Refactor evals package to expose `completion_fn`. #515

hwchung27 commented Mar 30, 2023

andrew-openai left a comment

andrew-openai Mar 31, 2023

andrew-openai Mar 31, 2023

andrew-openai Mar 31, 2023

jwang47 Apr 3, 2023

andrew-openai Apr 3, 2023

andrew-openai Apr 3, 2023 •

edited

andrew-openai Apr 3, 2023

jwang47 Apr 3, 2023

andrew-openai Apr 6, 2023

jwang47 Apr 10, 2023

jwang47 Apr 10, 2023

jwang47 Apr 10, 2023

jwang47 Apr 10, 2023

jwang47 Apr 10, 2023

jwang47 Apr 10, 2023

jwang47 Apr 11, 2023

jwang47 Apr 11, 2023

		@@ -97,6 +97,7 @@ def completion_query(
		return result, openai_create_prompt, metadata


		# TODO(hwc): remove this


		In this example, `gpt-3.5-turbo` is the model to evaluate, and `test-match` is the eval to run. The valid model names are those which you have access to via the API. The valid eval names are specified in the YAML files under `evals/registry/evals`, and their corresponding implementations can be found in `evals/elsuite`.
		In this example, `gpt-3.5-turbo` is an OpenAI model that we dynamically instantiate as a completion function using `OpenAIChatCompletionFn(model=gpt-3.5-turbo)`. Any implementation of the `CompletionFn` protocol can be run against `oaieval`. By default, we support calling `oaieval` with any model availableon the OpenAI API or with CompletionFunctions available in [`evals/registry/completion_fns`](../evals/registry/completion_fns/).

[evals] Refactor evals package to expose completion_fn. #515

[evals] Refactor evals package to expose completion_fn. #515

Conversation

hwchung27 commented Mar 30, 2023

Thank you for contributing an eval! ♥️

Eval details 📑

Eval name

Eval description

What makes this a useful eval?

Criteria for a good eval ✅

Unique eval value

Eval structure 🏗️

Final checklist 👀

Submission agreement

Email address validation

Limited availability acknowledgement

Submit eval

Eval JSON data

Eval

andrew-openai left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrew-openai Apr 3, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

[evals] Refactor evals package to expose `completion_fn`. #515

[evals] Refactor evals package to expose `completion_fn`. #515

andrew-openai Apr 3, 2023 •

edited