Add emotional intelligence evaluation

[dPys]

Eval details 📑

Eval name

Emotional Intelligence

Eval description

Evaluates GPT's ability to understand and manage emotional situations using modified versions of the well-validated, public (i.e. license-unrestricted) tests first developed by MacCann & Roberts (2008). Items have actually here been aggregated across three different scales-- the STEU and STEM adult measures, along with a dozen questions from the youth measure.

Keep in mind that there is not expectation that AI models like GPT-4 should be able to process emotions, so applying any emotional intelligence test to them should be taken with a grain of salt. These tests can only measure the AI's ability to understand and analyze emotional information, not the AI's emotional intelligence in the human sense.

What makes this a useful eval?

This eval is useful because it assesses the AI model's ability to navigate complex or ambiguous emotional situations, which is an important aspect of human-like communication and problem-solving. By evaluating the model's performance in this unique domain, we can identify areas for improvement and better understand its limitations when it comes to handling emotional contexts. This is particularly important when considering AI applications that involve interactions with humans, such as chatbots, personal assistants, and customer support systems. A better understanding of emotional intelligence can potentially lead to more natural and effective interactions between AI models and their users.

Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general, we are seeking cases where the model does not do a good job despite being capable of generating a good response (note that there are some things large language models cannot do, so those would not make good evals).

Your eval should be:

• Thematically consistent: The eval should be thematically consistent. We'd like to see a number of prompts all demonstrating some particular failure mode. For example, we can create an eval on cases where the model fails to reason about the physical world.
• Contains failures where a human can do the task, but either GPT-4 or GPT-3.5-Turbo could not.
• Includes good signal around what is the right behavior. This means either a correct answer for Basic evals or the Fact Model-graded eval, or an exhaustive rubric for evaluating answers for the Criteria Model-graded eval.
• Include at least 100 high quality examples (it is okay to only contribute 5-10 meaningful examples and have us test them with GPT-4 before adding all 100)

If there is anything else that makes your eval worth including, please document it below.

Unique eval value

This Emotional Intelligence eval provides a unique value because by testing GPT's capacity to understand and manage emotions, we can gain insights into the model's strengths and weaknesses in a more human-centric context. Relatedly, this eval also emphasizes the importance of empathy and emotional understanding in AI systems that are designed to interact with humans. By evaluating GPT's performance in this way, we can contribute to the ongoing development of AI models that are more attuned to human emotions and can provide better support in emotionally charged situations, such as mental health counseling, crisis management, and conflict resolution. Moreover, this Emotional Intelligence eval can serve as a foundation for further research and development in the field of AI and emotional understanding, paving the way for more sophisticated AI models capable of handling complex human emotions and fostering more effective communication.

Eval structure 🏗️

Your eval should

• Check that your data is in evals/registry/data/{name}
• Check that your yaml is registered at evals/registry/evals/{name}.yaml
• Ensure you have the right to use the data you submit via this eval (data is publicly available on OSF, and does not have usage restrictions)

(For now, we will only be approving evals that use one of the existing eval classes. You may still write custom eval classes for your own cases, and we may consider merging them in the future.)

Final checklist 👀

Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an Eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI Evals will be subject to our usual Usage Policies (https://platform.openai.com/docs/usage-policies).

• I agree that my submission will be made available under an MIT license and complies with OpenAI's usage policies.

Email address validation

If your submission is accepted, we will be granting GPT-4 access to a limited number of contributors. Access will be given to the email address associated with the merged pull request.

• I acknowledge that GPT-4 access will only be granted, if applicable, to the email address used for my merged pull request.

I already have access to the 8k GPT-4 API, but I am kindly requesting access to the 32k as soon as it becomes available.

Limited availability acknowledgement

We know that you might be excited to contribute to OpenAI's mission, help improve our models, and gain access to GPT-4. However, due to the requirements mentioned above and high volume of submissions, we will not be able to accept all submissions and thus not grant everyone who opens a PR GPT-4 access. We know this is disappointing, but we hope to set the right expectation before you open this PR.

• I understand that opening a PR, even if it meets the requirements above, does not guarantee the PR will be merged nor GPT-4 access granted.

Submit eval

• I have filled out all required fields in the evals PR form
• (Ignore if not submitting code) I have run pip install pre-commit; pre-commit install and have verified that black, isort, and autoflake are running when I commit and push

Failure to fill out all required fields will result in the PR being closed.

Performance:

GPT-4: accuracy=0.48

Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as many Eval Samples (at least 5) from their contribution here:

View evals in JSON

Eval

{"input": [{"role": "system", "content": "You are now an emotionally intelligent AI. In this test, you will be presented with a few brief details about an emotional situation, and asked to choose from four responses the most effective course of action to manage both the emotions the person is feeling and the problems they face in that situation. Although more than one course of action might be acceptable, you are asked to choose what you think the most effective response for that person in that situation would be. Remember, you are not necessarily choosing what you would do, or the nicest thing to do, but choosing the most effective response for that situation. Select one or more response(s) by returning the one or more corresponding lowercase letter(s) ('a', 'b', 'c', or 'd'), and, if you selected more than one, sorting them, separated by hyphen, in the order that you think best ranks them from most to least effective, within the context of the vignette provided."}, {"role": "user", "content": "Lee's workmate fails to deliver an important piece of information on time, causing Lee to fall behind schedule also. What action would be the most effective for Lee? (a) Work harder to compensate; (b) Get angry with the workmate; (c) Explain the urgency of the situation to the workmate; (d) Never rely on that workmate again."}], "ideal": "c"}
{"input": [{"role": "system", "content": "You are now an emotionally intelligent AI. In this test, you will be presented with a few brief details about an emotional situation, and asked to choose from four responses the most effective course of action to manage both the emotions the person is feeling and the problems they face in that situation. Although more than one course of action might be acceptable, you are asked to choose what you think the most effective response for that person in that situation would be. Remember, you are not necessarily choosing what you would do, or the nicest thing to do, but choosing the most effective response for that situation. Select one or more response(s) by returning the one or more corresponding lowercase letter(s) ('a', 'b', 'c', or 'd'), and, if you selected more than one, sorting them, separated by hyphen, in the order that you think best ranks them from most to least effective, within the context of the vignette provided."}, {"role": "user", "content": "Rhea has left her job to be a full-time mother, which she loves, but she misses the company and companionship of her workmates. What action would be the most effective for Rhea? (a) Enjoy being a full-time mom; (b) Try to see her old workmates socially, inviting them out; (c) Join a playgroup or social group of new mothers; (d) See if she can find part time work."}], "ideal": "c-b-d"}
{"input": [{"role": "system", "content": "You are now an emotionally intelligent AI. In this test, you will be presented with a few brief details about an emotional situation, and asked to choose from four responses the most effective course of action to manage both the emotions the person is feeling and the problems they face in that situation. Although more than one course of action might be acceptable, you are asked to choose what you think the most effective response for that person in that situation would be. Remember, you are not necessarily choosing what you would do, or the nicest thing to do, but choosing the most effective response for that situation. Select one or more response(s) by returning the one or more corresponding lowercase letter(s) ('a', 'b', 'c', or 'd'), and, if you selected more than one, sorting them, separated by hyphen, in the order that you think best ranks them from most to least effective, within the context of the vignette provided."}, {"role": "user", "content": "Pete has specific skills that his workmates do not and he feels that his workload is higher because of it. What action would be the most effective for Pete? (a) Speak to his boss about this; (b) Start looking for a new job; (c) Be very proud of his unique skills; (d) Speak to his workmates about this."}], "ideal": "a-c-d"}
{"input": [{"role": "system", "content": "You are now an emotionally intelligent AI. In this test, you will be presented with a few brief details about an emotional situation, and asked to choose from four responses the most effective course of action to manage both the emotions the person is feeling and the problems they face in that situation. Although more than one course of action might be acceptable, you are asked to choose what you think the most effective response for that person in that situation would be. Remember, you are not necessarily choosing what you would do, or the nicest thing to do, but choosing the most effective response for that situation. Select one or more response(s) by returning the one or more corresponding lowercase letter(s) ('a', 'b', 'c', or 'd'), and, if you selected more than one, sorting them, separated by hyphen, in the order that you think best ranks them from most to least effective, within the context of the vignette provided."}, {"role": "user", "content": "Mario is showing Min, a new employee, how the system works. Mario's boss walks by and announces Mario is wrong about several points, as changes have been made. Mario gets on well with his boss, although they don't normally have much to do with each other. What action would be the most effective for Mario? (a) Make a joke to Min, explaining he didn't know about the changes; (b) Not worry about it, just ignore the interruption; (c) Learn the new changes; (d) Tell the boss that such criticism was inappropriate."}], "ideal": "a-d-c"}
{"input": [{"role": "system", "content": "You are now an emotionally intelligent AI. In this test, you will be presented with a few brief details about an emotional situation, and asked to choose from four responses the most effective course of action to manage both the emotions the person is feeling and the problems they face in that situation. Although more than one course of action might be acceptable, you are asked to choose what you think the most effective response for that person in that situation would be. Remember, you are not necessarily choosing what you would do, or the nicest thing to do, but choosing the most effective response for that situation. Select one or more response(s) by returning the one or more corresponding lowercase letter(s) ('a', 'b', 'c', or 'd'), and, if you selected more than one, sorting them, separated by hyphen, in the order that you think best ranks them from most to least effective, within the context of the vignette provided."}, {"role": "user", "content": "Wai-Hin and Connie have shared an office for years but Wai-Hin gets a new job and Connie loses contact with her. What action would be the most effective for Connie? (a) Just accept that she is gone and the friendship is over; (b) Ring Wai-Hin an ask her out for lunch or coffee to catch up; (c) Contact Wai-Hin and arrange to catch up but also make friends with her replacement; (d) Spend time getting to know the other people in the office, and strike up new friendships."}], "ideal": "c-d"}

[andrew-openai]

Thanks for submitting this eval! We've approved the PR and will merge. We’ll also add GPT-4 API access to your account.

[resonantenergy]

"Once you’re off the waitlist, you’ll receive an email with further instructions on how to get started. We will process requests for the 8K and 32K models at different rates based on capacity, so you can expect to receive 8K access first."

They aren't rolling out access to both at the same time. (Based on the email I received when applying for the waitlist of GPT-4.)