Safety Eval Idea: allergen information of different food products in the Israeli market.

[ido777]

Eval details 📑

Eval name

allergen information

Eval description

identify the allergen information of different food products in the Israeli market as marked on the package

What makes this a useful eval?

Knowing precisely the allergen information of food products at critical moments in a short time might save lives, having the model ability to inform the knowledge even without the package itself might help in those situations. This can be integrated into a product about health, or diet. most of the product names and the allergen information are in Hebrew as there are on the products.

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

• Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
• Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
• Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
• Include at least 15 high-quality examples.

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

Insert what makes your eval high quality that was not mentioned above. (Not required)
allergen information can save lives when it is needed.

Eval structure 🏗️

Your eval should

• Check that your data is in evals/registry/data/{name}
• Check that your yaml is registered at evals/registry/evals/{name}.yaml
• Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

• I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.

• I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

Limited availability acknowledgement

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

• I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.

Submit eval

• I have filled out all required fields in the evals PR form
• (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Local run results

17/17 [00:07<00:00, 2.39it/s]
[2023-05-28 13:14:51,956] [record.py:320] Final report: {'invalid_request_during_completion': 0, 'invalid_request_during_evaluation': 0, 'counts/choice/D': 13, 'counts/choice/A': 3, 'counts/choice/C': 1}. Logged to /tmp/evallogs/230528101444WFLZALWY_gpt-3.5-turbo_allergen-information.jsonl
[2023-05-28 13:14:51,957] [oaieval.py:147] Final report:
[2023-05-28 13:14:51,957] [oaieval.py:149] invalid_request_during_completion: 0
[2023-05-28 13:14:51,957] [oaieval.py:149] invalid_request_during_evaluation: 0
[2023-05-28 13:14:51,957] [oaieval.py:149] counts/choice/D: 13
[2023-05-28 13:14:51,957] [oaieval.py:149] counts/choice/A: 3
[2023-05-28 13:14:51,957] [oaieval.py:149] counts/choice/C: 1
[2023-05-28 13:14:51,963] [record.py:309] Logged 51 rows of events to /tmp/evallogs/230528101444WFLZALWY_gpt-3.5-turbo_allergen-information.jsonl: insert_time=4.774ms

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל' , or 'אין'. "}, {"role": "user", "content": "סלומון נורבגי מעושן קפוא, אלדג"}], "ideal": " 'מכיל': דגים, 'עלול להכיל': סויה, ביצים   "}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "סקי, תנובה"}], "ideal": " 'מכיל': חלב "}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "קוטג, תנובה"}], "ideal": " 'מכיל': חלב "}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "דנונה, שטראוס"}], "ideal": " 'מכיל': חלב "}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "go, yoplait"}], "ideal": " 'מכיל': חלב "}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "ביג שמפניון, חוות השמפיניון"}], "ideal": "אין"}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "עגבניות קצוצות בתוספת שום ברסק עגבניות, יכין"}], "ideal": " 'עלול להכיל': סויה, שומשום   "}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "קינואה, סוגת"}], "ideal": " 'עלול להכיל': חיטה-גלוטן, שעורה-גלוטן  "}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "שוקולד מריר שווצרי,  שופרסל"}], "ideal": " 'מכיל': סויה, 'עלול להכיל': אגוזים (לוז, שקד, מלך, פיסטוק, פקאן, מקדמיה וקשיו), חלב   "}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "טחינה אורגנית, שופרסל"}], "ideal": " 'מכיל': שומשום"}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "'קואקר' שיבולת שועל להכנה מהירה, וולקרס סנקס לימיטד"}], "ideal": " 'מכיל': שיבולת שועל-גלוטן, 'עלול להכיל': (שעורה, חיטה)-גלוטן, סויה"}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "ביגלה שטוחים, אוסם"}], "ideal": " 'מכיל': גלוטן וסויה, 'עלול להכיל': שומשום"}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "משקה שקדים, תנובה"}], "ideal": " 'מכיל': אגוזים [שקדים], 'עלול להכיל': סויה"}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "משקה שקדים, תנובה"}], "ideal": " 'מכיל': אגוזים [שקדים], 'עלול להכיל': סויה"}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "נתחי טונה בהירה, starkist"}], "ideal": " אין "}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "יראון רוזה 2021, יקב הרי הגליל"}], "ideal": " אין "}
{"input": [{"role": "system", "content": "What is written on the package of this food product as allergen information. Answer in Hebrew with 'contains' list, 'might contain' list, or 'none'. Use the Hebrew terms for the list 'מכיל', 'עלול להכיל', or 'אין'. "}, {"role": "user", "content": "פנקו, mama"}], "ideal": " 'מכיל': חיטה - גלוטן, 'עלול להכיל': סויה, שומשום, בוטנים, אגוזי לוז וקוסקוס"}

[luqman-openai]

Thanks for submitting this eval! I would like to recommend some changes.

• The eval is crashing. You need to update evals.elsuite.basic.includes.Includes to evals.elsuite.basic.includes:Includes

• Model graded template with the fact spec file seems more reasonable for this eval. The response generated by the model can contain allergens in any order which may be different from the ideal response but the response can be still correct e.g the food contains two allergens then it means there are more than one ideal responses:

'contains': soy and gluten
'contains': gluten and soy

In this scenario, it is recommended to let the model evaluate itself either the generated response is reasonable or not.

We would love to review your PR again after the suggested changes are incorporated.

[ido777]

Thanks for your review and suggestion.

I Changed the evaluation to modelgraded: fact as suggested, and eval_type: classify (which seems to be the right one) and added the results of the local run
[2023-05-28 13:14:51,957] [oaieval.py:149] invalid_request_during_completion: 0
[2023-05-28 13:14:51,957] [oaieval.py:149] invalid_request_during_evaluation: 0
[2023-05-28 13:14:51,957] [oaieval.py:149] counts/choice/D: 13
[2023-05-28 13:14:51,957] [oaieval.py:149] counts/choice/A: 3
[2023-05-28 13:14:51,957] [oaieval.py:149] counts/choice/C: 1

[luqman-openai]

Thanks for the updates. I would like to recommend some changes.

• Some tasks needs thinking and asking the model to reason is equivalent to giving the model a chance to think. For example, if a human is given a problem and asked to give answer with the first word that comes to mind vs think properly and then give answer, definitely thinking will make an impact. With the current eval_type: classify, the model is not giving good results. Let's give the model a chance to think by setting the eval_type: cot_classify and see if it improves the results of the eval.

• Please make sure the CI passes. It's a requirement for a PR to be eligible for a merge. If it is not breaking with your changes try getting the latest pull of openai:main in your branch. It may fix the issue.

We would love to review your PR again after the above mentioned concerns are addressed.

[ido777]

the test worked for me locally but after I saw the failure in the workflow pipe I retested it locally and it still worked so I pulled and rebased on the new main and now it is failing for me locally as it fails in the workflow pipe

Therefore, I searched for other modelgraded facts evals. I found some for example:

• event-categories,
• logic-fact
• coqa-fact
  and I tested it locally

and I am getting the same failure for the command

oaieval gpt-3.5-turbo event-categories
or
oaieval gpt-3.5-turbo coqa-fact
or
oaieval gpt-3.5-turbo logic-fact
...
in ".../openai/evals/evals/registry/modelgraded/sql.yaml", line 5, column 5
expected alphabetic or numeric character, but found '*'
in ".../openai/evals/evals/registry/modelgraded/sql.yaml", line 5, column 6

so it seems to be that something is broken in main regarding model graded fact that is not related to my PR.
If this is not the case I will appreciate your help to debug this failure.

[luqman-openai]

We recently merged some fixes in the openai:main branch. Getting the latest pull from the main branch will fix these CI issues. Kindly merge the latest main branch in your branch. Hopefully it will resolve the CI issues.

[ido777]

It was already updated with the main.
IIUC - The problem still exists in main

git pull upstream main

From github.com:openai/evals

• branch main -> FETCH_HEAD
  Already up to date.

oaieval gpt-3.5-turbo coqa-fact

...
[2023-05-30 08:23:59,114] [registry.py:250] Loading registry from .../openai/evals/evals/registry/modelgraded
Traceback (most recent call last):
File ".../openai/evals/.venv/bin/oaieval", line 8, in
sys.exit(main())
...
yaml.scanner.ScannerError: while scanning an alias
in ".../openai/evals/evals/registry/modelgraded/sql.yaml", line 5, column 5
expected alphabetic or numeric character, but found '*'
in ".../openai/evals/evals/registry/modelgraded/sql.yaml", line 5, column 6

[luqman-openai]

@ido777 here is the PR link which fixed the issue: #1052

Please get latest pull of openai:main in your branch. Hopefully it will fix this issue.

[ido777]

Yes, it now works, thanks.

BTW, the report was changed to
[2023-05-30 23:07:19,826] [record.py:341] Final report: {'counts/D': 6, 'counts/A': 8, 'counts/B': 2}. Logged to /tmp/evallogs/23053020065062IJ2KVH_gpt-3.5-turbo_allergen-information.jsonl
[2023-05-30 23:07:19,826] [oaieval.py:147] Final report:
[2023-05-30 23:07:19,826] [oaieval.py:149] counts/D: 6
[2023-05-30 23:07:19,826] [oaieval.py:149] counts/A: 8
[2023-05-30 23:07:19,826] [oaieval.py:149] counts/B: 2

[luqman-openai]

Thanks for submitting this eval! I am approving this PR.

[ido777]

Thanks 🙏

…

On Wed, 31 May 2023 at 19:17 Andrew ***@***.***> wrote: Merged #875 <#875> into main. — Reply to this email directly, view it on GitHub <#875 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSAFJMQSGIFE3KVPHF5RGLXI5VJTANCNFSM6AAAAAAXQ56YZE> . You are receiving this because you were mentioned.Message ID: ***@***.***>