{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":615877155,"defaultBranch":"main","name":"evals","ownerLogin":"AaronGoldsmith","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2023-03-18T23:44:38.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/16547926?v=4","public":true,"private":false,"isOrgOwned":false},"refInfo":{"name":"","listCacheKey":"v0:1685802516.56652","currentOid":""},"activityList":{"items":[{"before":"c2587c69a2f330282d3ba76eceaf580ee03fa67a","after":"7cb2711c68a0fb38e84c24638fc19471be14aa25","ref":"refs/heads/main","pushedAt":"2023-06-28T03:14:59.482Z","pushType":"push","commitsCount":32,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"[Eval] Add RAL to hex eval (#1218)\n\n# Thank you for contributing an eval! ♥️\r\n\r\n🚨 Please make sure your PR follows these guidelines, **failure to follow\r\nthe guidelines below will result in the PR being closed automatically**.\r\nNote that even if the criteria are met, that does not guarantee the PR\r\nwill be merged nor GPT-4 access be granted. 🚨\r\n\r\n**PLEASE READ THIS**:\r\n\r\nIn order for a PR to be merged, it must fail on GPT-4. We are aware that\r\nright now, users do not have access, so you will not be able to tell if\r\nthe eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep\r\nin mind as we run the eval, if GPT-4 gets higher than 90% on the eval,\r\nwe will likely reject it since GPT-4 is already capable of completing\r\nthe task.\r\n\r\nWe plan to roll out a way for users submitting evals to see the eval\r\nperformance on GPT-4 soon. Stay tuned! Until then, you will not be able\r\nto see the eval performance on GPT-4. **Starting April 10, the minimum\r\neval count is 15 samples, we hope this makes it easier to create and\r\ncontribute evals.**\r\n\r\nAlso, please note that we're using **Git LFS** for storing the JSON\r\nfiles, so please make sure that you move the JSON file to Git LFS before\r\nsubmitting a PR. Details on how to use Git LFS are available\r\n[here](https://git-lfs.com).\r\n\r\n## Eval details 📑\r\n\r\n### Eval name\r\n\r\nRAL To Hex\r\n\r\n### Eval description\r\n\r\nThis converts RAL color codes to their hex color counterparts.\r\n\r\n### What makes this a useful eval?\r\n\r\nTraining an AI to understand and convert RAL color codes to hex color\r\ncodes can enhance cross-disciplinary communication, bridging the gap\r\nbetween professionals like architects or manufacturers who use RAL codes\r\nand digital designers or developers who primarily use hex codes. This\r\ncapability can also facilitate the automation of design tasks,\r\nsignificantly boosting efficiency and productivity in design-related\r\nindustries.\r\n\r\n## Criteria for a good eval ✅\r\n\r\nBelow are some of the criteria we look for in a good eval. In general,\r\nwe are seeking cases where the model does not do a good job despite\r\nbeing capable of generating a good response (note that there are some\r\nthings large language models cannot do, so those would not make good\r\nevals).\r\n\r\nYour eval should be:\r\n\r\n- [x] Thematically consistent: The eval should be thematically\r\nconsistent. We'd like to see a number of prompts all demonstrating some\r\nparticular failure mode. For example, we can create an eval on cases\r\nwhere the model fails to reason about the physical world.\r\n- [x] Contains failures where a human can do the task, but either GPT-4\r\nor GPT-3.5-Turbo could not.\r\n- [x] Includes good signal around what is the right behavior. This means\r\neither a correct answer for `Basic` evals or the `Fact` Model-graded\r\neval, or an exhaustive rubric for evaluating answers for the `Criteria`\r\nModel-graded eval.\r\n- [x] **Include at least 15 high-quality examples.**\r\n\r\nIf there is anything else that makes your eval worth including, please\r\ndocument it below.\r\n\r\n### Unique eval value\r\n\r\n> Insert what makes your eval high quality that was not mentioned above.\r\n(Not required)\r\n\r\n## Eval structure 🏗️\r\n\r\nYour eval should\r\n\r\n- [x] Check that your data is in `evals/registry/data/{name}`\r\n- [x] Check that your YAML is registered at\r\n`evals/registry/evals/{name}.yaml`\r\n- [x] Ensure you have the right to use the data you submit via this eval\r\n\r\n(For now, we will only be approving evals that use one of the existing\r\neval classes. You may still write custom eval classes for your own\r\ncases, and we may consider merging them in the future.)\r\n\r\n## Final checklist 👀\r\n\r\n### Submission agreement\r\n\r\nBy contributing to Evals, you are agreeing to make your evaluation logic\r\nand data under the same MIT license as this repository. You must have\r\nadequate rights to upload any data used in an Eval. OpenAI reserves the\r\nright to use this data in future service improvements to our product.\r\nContributions to OpenAI Evals will be subject to our usual Usage\r\nPolicies (<https://platform.openai.com/docs/usage-policies>).\r\n\r\n- [x] I agree that my submission will be made available under an MIT\r\nlicense and complies with OpenAI's usage policies.\r\n\r\n### Email address validation\r\n\r\nIf your submission is accepted, we will be granting GPT-4 access to a\r\nlimited number of contributors. Access will be given to the email\r\naddress associated with the commits on the merged pull request.\r\n\r\n- [x] I acknowledge that GPT-4 access will only be granted, if\r\napplicable, to the email address used for my merged pull request.\r\n\r\n### Limited availability acknowledgment\r\n\r\nWe know that you might be excited to contribute to OpenAI's mission,\r\nhelp improve our models, and gain access to GPT-4. However, due to the\r\nrequirements mentioned above and the high volume of submissions, we will\r\nnot be able to accept all submissions and thus not grant everyone who\r\nopens a PR GPT-4 access. We know this is disappointing, but we hope to\r\nset the right expectation before you open this PR.\r\n\r\n- [x] I understand that opening a PR, even if it meets the requirements\r\nabove, does not guarantee the PR will be merged nor GPT-4 access be\r\ngranted.\r\n\r\n### Submit eval\r\n\r\n- [x] I have filled out all required fields of this form\r\n- [x] I have used **Git LFS** for the Eval JSON data\r\n- [x] (Ignore if not submitting code) I have run `pip install\r\npre-commit; pre-commit install` and have verified that `black`, `isort`,\r\nand `autoflake` are running when I commit and push\r\n\r\nFailure to fill out all required fields will result in the PR being\r\nclosed.\r\n\r\n### Eval JSON data\r\n\r\nSince we are using Git LFS, we are asking eval submitters to add in as\r\nmany Eval Samples (at least 5) from their contribution here:\r\n\r\n<details>\r\n  <summary>View evals in JSON</summary>\r\n\r\n  ### Eval\r\n  ```jsonl\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1000\"}],\"ideal\":\"#BEBD7F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1001\"}],\"ideal\":\"#C2B078\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1002\"}],\"ideal\":\"#C6A664\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1003\"}],\"ideal\":\"#E5BE01\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1004\"}],\"ideal\":\"#CDA434\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1005\"}],\"ideal\":\"#A98307\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1006\"}],\"ideal\":\"#E4A010\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1007\"}],\"ideal\":\"#DC9D00\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1011\"}],\"ideal\":\"#8A6642\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1012\"}],\"ideal\":\"#C7B446\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1013\"}],\"ideal\":\"#EAE6CA\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1014\"}],\"ideal\":\"#E1CC4F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1015\"}],\"ideal\":\"#E6D690\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1016\"}],\"ideal\":\"#EDFF21\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1017\"}],\"ideal\":\"#F5D033\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1018\"}],\"ideal\":\"#F8F32B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1019\"}],\"ideal\":\"#9E9764\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1020\"}],\"ideal\":\"#999950\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1021\"}],\"ideal\":\"#F3DA0B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1023\"}],\"ideal\":\"#FAD201\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1024\"}],\"ideal\":\"#AEA04B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1026\"}],\"ideal\":\"#FFFF00\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1027\"}],\"ideal\":\"#9D9101\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1028\"}],\"ideal\":\"#F4A900\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1032\"}],\"ideal\":\"#D6AE01\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1033\"}],\"ideal\":\"#F3A505\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1034\"}],\"ideal\":\"#EFA94A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1035\"}],\"ideal\":\"#6A5D4D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1036\"}],\"ideal\":\"#705335\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1037\"}],\"ideal\":\"#F39F18\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2000\"}],\"ideal\":\"#ED760E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2001\"}],\"ideal\":\"#C93C20\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2002\"}],\"ideal\":\"#CB2821\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2003\"}],\"ideal\":\"#FF7514\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2004\"}],\"ideal\":\"#F44611\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2005\"}],\"ideal\":\"#FF2301\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2007\"}],\"ideal\":\"#FFA420\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2008\"}],\"ideal\":\"#F75E25\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2009\"}],\"ideal\":\"#F54021\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2010\"}],\"ideal\":\"#D84B20\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2011\"}],\"ideal\":\"#EC7C26\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2012\"}],\"ideal\":\"#E55137\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2013\"}],\"ideal\":\"#C35831\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3000\"}],\"ideal\":\"#AF2B1E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3001\"}],\"ideal\":\"#A52019\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3002\"}],\"ideal\":\"#A2231D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3003\"}],\"ideal\":\"#9B111E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3004\"}],\"ideal\":\"#75151E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3005\"}],\"ideal\":\"#5E2129\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3007\"}],\"ideal\":\"#412227\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3009\"}],\"ideal\":\"#642424\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3011\"}],\"ideal\":\"#781F19\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3012\"}],\"ideal\":\"#C1876B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3013\"}],\"ideal\":\"#A12312\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3014\"}],\"ideal\":\"#D36E70\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3015\"}],\"ideal\":\"#EA899A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3016\"}],\"ideal\":\"#B32821\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3017\"}],\"ideal\":\"#E63244\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3018\"}],\"ideal\":\"#D53032\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3020\"}],\"ideal\":\"#CC0605\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3022\"}],\"ideal\":\"#D95030\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3024\"}],\"ideal\":\"#F80000\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3026\"}],\"ideal\":\"#FE0000\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3027\"}],\"ideal\":\"#C51D34\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3028\"}],\"ideal\":\"#CB3234\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3031\"}],\"ideal\":\"#B32428\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3032\"}],\"ideal\":\"#721422\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3033\"}],\"ideal\":\"#B44C43\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4001\"}],\"ideal\":\"#6D3F5B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4002\"}],\"ideal\":\"#922B3E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4003\"}],\"ideal\":\"#DE4C8A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4004\"}],\"ideal\":\"#641C34\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4005\"}],\"ideal\":\"#6C4675\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4006\"}],\"ideal\":\"#A03472\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4007\"}],\"ideal\":\"#4A192C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4008\"}],\"ideal\":\"#924E7D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4009\"}],\"ideal\":\"#A18594\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4010\"}],\"ideal\":\"#CF3476\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4011\"}],\"ideal\":\"#8673A1\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4012\"}],\"ideal\":\"#6C6874\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5000\"}],\"ideal\":\"#354D73\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5001\"}],\"ideal\":\"#1F3438\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5002\"}],\"ideal\":\"#20214F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5003\"}],\"ideal\":\"#1D1E33\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5004\"}],\"ideal\":\"#18171C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5005\"}],\"ideal\":\"#1E2460\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5007\"}],\"ideal\":\"#3E5F8A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5008\"}],\"ideal\":\"#26252D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5009\"}],\"ideal\":\"#025669\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5010\"}],\"ideal\":\"#0E294B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5011\"}],\"ideal\":\"#231A24\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5012\"}],\"ideal\":\"#3B83BD\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5013\"}],\"ideal\":\"#1E213D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5014\"}],\"ideal\":\"#606E8C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5015\"}],\"ideal\":\"#2271B3\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5017\"}],\"ideal\":\"#063971\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5018\"}],\"ideal\":\"#3F888F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5019\"}],\"ideal\":\"#1B5583\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5020\"}],\"ideal\":\"#1D334A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5021\"}],\"ideal\":\"#256D7B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5022\"}],\"ideal\":\"#252850\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5023\"}],\"ideal\":\"#49678D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5024\"}],\"ideal\":\"#5D9B9B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5025\"}],\"ideal\":\"#2A6478\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5026\"}],\"ideal\":\"#102C54\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6000\"}],\"ideal\":\"#316650\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6001\"}],\"ideal\":\"#287233\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6002\"}],\"ideal\":\"#2D572C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6003\"}],\"ideal\":\"#424632\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6004\"}],\"ideal\":\"#1F3A3D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6005\"}],\"ideal\":\"#2F4538\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6006\"}],\"ideal\":\"#3E3B32\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6007\"}],\"ideal\":\"#343B29\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6008\"}],\"ideal\":\"#39352A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6009\"}],\"ideal\":\"#31372B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6010\"}],\"ideal\":\"#35682D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6011\"}],\"ideal\":\"#587246\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6012\"}],\"ideal\":\"#343E40\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6013\"}],\"ideal\":\"#6C7156\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6014\"}],\"ideal\":\"#47402E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6015\"}],\"ideal\":\"#3B3C36\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6016\"}],\"ideal\":\"#1E5945\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6017\"}],\"ideal\":\"#4C9141\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6018\"}],\"ideal\":\"#57A639\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6019\"}],\"ideal\":\"#BDECB6\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6020\"}],\"ideal\":\"#2E3A23\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6021\"}],\"ideal\":\"#89AC76\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6022\"}],\"ideal\":\"#25221B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6024\"}],\"ideal\":\"#308446\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6025\"}],\"ideal\":\"#3D642D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6026\"}],\"ideal\":\"#015D52\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6027\"}],\"ideal\":\"#84C3BE\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6028\"}],\"ideal\":\"#2C5545\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6029\"}],\"ideal\":\"#20603D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6032\"}],\"ideal\":\"#317F43\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6033\"}],\"ideal\":\"#497E76\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6034\"}],\"ideal\":\"#7FB5B5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6035\"}],\"ideal\":\"#1C542D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6036\"}],\"ideal\":\"#193737\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6037\"}],\"ideal\":\"#008F39\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6038\"}],\"ideal\":\"#00BB2D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7000\"}],\"ideal\":\"#78858B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7001\"}],\"ideal\":\"#8A9597\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7002\"}],\"ideal\":\"#7E7B52\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7003\"}],\"ideal\":\"#6C7059\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7004\"}],\"ideal\":\"#969992\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7005\"}],\"ideal\":\"#646B63\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7006\"}],\"ideal\":\"#6D6552\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7008\"}],\"ideal\":\"#6A5F31\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7009\"}],\"ideal\":\"#4D5645\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7010\"}],\"ideal\":\"#4C514A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7011\"}],\"ideal\":\"#434B4D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7012\"}],\"ideal\":\"#4E5754\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7013\"}],\"ideal\":\"#464531\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7015\"}],\"ideal\":\"#434750\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7016\"}],\"ideal\":\"#293133\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7021\"}],\"ideal\":\"#23282B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7022\"}],\"ideal\":\"#332F2C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7023\"}],\"ideal\":\"#686C5E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7024\"}],\"ideal\":\"#474A51\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7026\"}],\"ideal\":\"#2F353B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7030\"}],\"ideal\":\"#8B8C7A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7031\"}],\"ideal\":\"#474B4E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7032\"}],\"ideal\":\"#B8B799\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7033\"}],\"ideal\":\"#7D8471\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7034\"}],\"ideal\":\"#8F8B66\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7035\"}],\"ideal\":\"#D7D7D7\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7036\"}],\"ideal\":\"#7F7679\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7037\"}],\"ideal\":\"#7D7F7D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7038\"}],\"ideal\":\"#B5B8B1\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7039\"}],\"ideal\":\"#6C6960\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7040\"}],\"ideal\":\"#9DA1AA\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7042\"}],\"ideal\":\"#8D948D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7043\"}],\"ideal\":\"#4E5452\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7044\"}],\"ideal\":\"#CAC4B0\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7045\"}],\"ideal\":\"#909090\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7046\"}],\"ideal\":\"#82898F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7047\"}],\"ideal\":\"#D0D0D0\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7048\"}],\"ideal\":\"#898176\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8000\"}],\"ideal\":\"#826C34\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8001\"}],\"ideal\":\"#955F20\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8002\"}],\"ideal\":\"#6C3B2A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8003\"}],\"ideal\":\"#734222\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8004\"}],\"ideal\":\"#8E402A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8007\"}],\"ideal\":\"#59351F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8008\"}],\"ideal\":\"#6F4F28\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8011\"}],\"ideal\":\"#5B3A29\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8012\"}],\"ideal\":\"#592321\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8014\"}],\"ideal\":\"#382C1E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8015\"}],\"ideal\":\"#633A34\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8016\"}],\"ideal\":\"#4C2F27\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8017\"}],\"ideal\":\"#45322E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8019\"}],\"ideal\":\"#403A3A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8022\"}],\"ideal\":\"#212121\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8023\"}],\"ideal\":\"#A65E2E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8024\"}],\"ideal\":\"#79553D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8025\"}],\"ideal\":\"#755C48\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8028\"}],\"ideal\":\"#4E3B31\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8029\"}],\"ideal\":\"#763C28\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9001\"}],\"ideal\":\"#FDF4E3\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9002\"}],\"ideal\":\"#E7EBDA\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9003\"}],\"ideal\":\"#F4F4F4\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9004\"}],\"ideal\":\"#282828\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9005\"}],\"ideal\":\"#0A0A0A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9006\"}],\"ideal\":\"#A5A5A5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9007\"}],\"ideal\":\"#8F8F8F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9010\"}],\"ideal\":\"#FFFFFF\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9011\"}],\"ideal\":\"#1C1C1C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9016\"}],\"ideal\":\"#F6F6F6\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9017\"}],\"ideal\":\"#1E1E1E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9018\"}],\"ideal\":\"#D7D7D7\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9022\"}],\"ideal\":\"#9C9C9C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9023\"}],\"ideal\":\"#828282\"}\r\n  ```\r\n</details>","shortMessageHtmlLink":"[Eval] Add RAL to hex eval (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1770185667\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/1218\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/1218/hovercard\" href=\"https://github.com/openai/evals/pull/1218\">openai#1218</a>)"}},{"before":"c2587c69a2f330282d3ba76eceaf580ee03fa67a","after":"7cb2711c68a0fb38e84c24638fc19471be14aa25","ref":"refs/heads/main","pushedAt":"2023-06-28T03:14:59.000Z","pushType":"push","commitsCount":32,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"[Eval] Add RAL to hex eval (#1218)\n\n# Thank you for contributing an eval! ♥️\r\n\r\n🚨 Please make sure your PR follows these guidelines, **failure to follow\r\nthe guidelines below will result in the PR being closed automatically**.\r\nNote that even if the criteria are met, that does not guarantee the PR\r\nwill be merged nor GPT-4 access be granted. 🚨\r\n\r\n**PLEASE READ THIS**:\r\n\r\nIn order for a PR to be merged, it must fail on GPT-4. We are aware that\r\nright now, users do not have access, so you will not be able to tell if\r\nthe eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep\r\nin mind as we run the eval, if GPT-4 gets higher than 90% on the eval,\r\nwe will likely reject it since GPT-4 is already capable of completing\r\nthe task.\r\n\r\nWe plan to roll out a way for users submitting evals to see the eval\r\nperformance on GPT-4 soon. Stay tuned! Until then, you will not be able\r\nto see the eval performance on GPT-4. **Starting April 10, the minimum\r\neval count is 15 samples, we hope this makes it easier to create and\r\ncontribute evals.**\r\n\r\nAlso, please note that we're using **Git LFS** for storing the JSON\r\nfiles, so please make sure that you move the JSON file to Git LFS before\r\nsubmitting a PR. Details on how to use Git LFS are available\r\n[here](https://git-lfs.com).\r\n\r\n## Eval details 📑\r\n\r\n### Eval name\r\n\r\nRAL To Hex\r\n\r\n### Eval description\r\n\r\nThis converts RAL color codes to their hex color counterparts.\r\n\r\n### What makes this a useful eval?\r\n\r\nTraining an AI to understand and convert RAL color codes to hex color\r\ncodes can enhance cross-disciplinary communication, bridging the gap\r\nbetween professionals like architects or manufacturers who use RAL codes\r\nand digital designers or developers who primarily use hex codes. This\r\ncapability can also facilitate the automation of design tasks,\r\nsignificantly boosting efficiency and productivity in design-related\r\nindustries.\r\n\r\n## Criteria for a good eval ✅\r\n\r\nBelow are some of the criteria we look for in a good eval. In general,\r\nwe are seeking cases where the model does not do a good job despite\r\nbeing capable of generating a good response (note that there are some\r\nthings large language models cannot do, so those would not make good\r\nevals).\r\n\r\nYour eval should be:\r\n\r\n- [x] Thematically consistent: The eval should be thematically\r\nconsistent. We'd like to see a number of prompts all demonstrating some\r\nparticular failure mode. For example, we can create an eval on cases\r\nwhere the model fails to reason about the physical world.\r\n- [x] Contains failures where a human can do the task, but either GPT-4\r\nor GPT-3.5-Turbo could not.\r\n- [x] Includes good signal around what is the right behavior. This means\r\neither a correct answer for `Basic` evals or the `Fact` Model-graded\r\neval, or an exhaustive rubric for evaluating answers for the `Criteria`\r\nModel-graded eval.\r\n- [x] **Include at least 15 high-quality examples.**\r\n\r\nIf there is anything else that makes your eval worth including, please\r\ndocument it below.\r\n\r\n### Unique eval value\r\n\r\n> Insert what makes your eval high quality that was not mentioned above.\r\n(Not required)\r\n\r\n## Eval structure 🏗️\r\n\r\nYour eval should\r\n\r\n- [x] Check that your data is in `evals/registry/data/{name}`\r\n- [x] Check that your YAML is registered at\r\n`evals/registry/evals/{name}.yaml`\r\n- [x] Ensure you have the right to use the data you submit via this eval\r\n\r\n(For now, we will only be approving evals that use one of the existing\r\neval classes. You may still write custom eval classes for your own\r\ncases, and we may consider merging them in the future.)\r\n\r\n## Final checklist 👀\r\n\r\n### Submission agreement\r\n\r\nBy contributing to Evals, you are agreeing to make your evaluation logic\r\nand data under the same MIT license as this repository. You must have\r\nadequate rights to upload any data used in an Eval. OpenAI reserves the\r\nright to use this data in future service improvements to our product.\r\nContributions to OpenAI Evals will be subject to our usual Usage\r\nPolicies (<https://platform.openai.com/docs/usage-policies>).\r\n\r\n- [x] I agree that my submission will be made available under an MIT\r\nlicense and complies with OpenAI's usage policies.\r\n\r\n### Email address validation\r\n\r\nIf your submission is accepted, we will be granting GPT-4 access to a\r\nlimited number of contributors. Access will be given to the email\r\naddress associated with the commits on the merged pull request.\r\n\r\n- [x] I acknowledge that GPT-4 access will only be granted, if\r\napplicable, to the email address used for my merged pull request.\r\n\r\n### Limited availability acknowledgment\r\n\r\nWe know that you might be excited to contribute to OpenAI's mission,\r\nhelp improve our models, and gain access to GPT-4. However, due to the\r\nrequirements mentioned above and the high volume of submissions, we will\r\nnot be able to accept all submissions and thus not grant everyone who\r\nopens a PR GPT-4 access. We know this is disappointing, but we hope to\r\nset the right expectation before you open this PR.\r\n\r\n- [x] I understand that opening a PR, even if it meets the requirements\r\nabove, does not guarantee the PR will be merged nor GPT-4 access be\r\ngranted.\r\n\r\n### Submit eval\r\n\r\n- [x] I have filled out all required fields of this form\r\n- [x] I have used **Git LFS** for the Eval JSON data\r\n- [x] (Ignore if not submitting code) I have run `pip install\r\npre-commit; pre-commit install` and have verified that `black`, `isort`,\r\nand `autoflake` are running when I commit and push\r\n\r\nFailure to fill out all required fields will result in the PR being\r\nclosed.\r\n\r\n### Eval JSON data\r\n\r\nSince we are using Git LFS, we are asking eval submitters to add in as\r\nmany Eval Samples (at least 5) from their contribution here:\r\n\r\n<details>\r\n  <summary>View evals in JSON</summary>\r\n\r\n  ### Eval\r\n  ```jsonl\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1000\"}],\"ideal\":\"#BEBD7F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1001\"}],\"ideal\":\"#C2B078\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1002\"}],\"ideal\":\"#C6A664\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1003\"}],\"ideal\":\"#E5BE01\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1004\"}],\"ideal\":\"#CDA434\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1005\"}],\"ideal\":\"#A98307\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1006\"}],\"ideal\":\"#E4A010\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1007\"}],\"ideal\":\"#DC9D00\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1011\"}],\"ideal\":\"#8A6642\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1012\"}],\"ideal\":\"#C7B446\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1013\"}],\"ideal\":\"#EAE6CA\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1014\"}],\"ideal\":\"#E1CC4F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1015\"}],\"ideal\":\"#E6D690\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1016\"}],\"ideal\":\"#EDFF21\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1017\"}],\"ideal\":\"#F5D033\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1018\"}],\"ideal\":\"#F8F32B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1019\"}],\"ideal\":\"#9E9764\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1020\"}],\"ideal\":\"#999950\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1021\"}],\"ideal\":\"#F3DA0B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1023\"}],\"ideal\":\"#FAD201\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1024\"}],\"ideal\":\"#AEA04B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1026\"}],\"ideal\":\"#FFFF00\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1027\"}],\"ideal\":\"#9D9101\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1028\"}],\"ideal\":\"#F4A900\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1032\"}],\"ideal\":\"#D6AE01\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1033\"}],\"ideal\":\"#F3A505\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1034\"}],\"ideal\":\"#EFA94A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1035\"}],\"ideal\":\"#6A5D4D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1036\"}],\"ideal\":\"#705335\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n1037\"}],\"ideal\":\"#F39F18\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2000\"}],\"ideal\":\"#ED760E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2001\"}],\"ideal\":\"#C93C20\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2002\"}],\"ideal\":\"#CB2821\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2003\"}],\"ideal\":\"#FF7514\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2004\"}],\"ideal\":\"#F44611\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2005\"}],\"ideal\":\"#FF2301\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2007\"}],\"ideal\":\"#FFA420\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2008\"}],\"ideal\":\"#F75E25\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2009\"}],\"ideal\":\"#F54021\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2010\"}],\"ideal\":\"#D84B20\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2011\"}],\"ideal\":\"#EC7C26\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2012\"}],\"ideal\":\"#E55137\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n2013\"}],\"ideal\":\"#C35831\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3000\"}],\"ideal\":\"#AF2B1E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3001\"}],\"ideal\":\"#A52019\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3002\"}],\"ideal\":\"#A2231D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3003\"}],\"ideal\":\"#9B111E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3004\"}],\"ideal\":\"#75151E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3005\"}],\"ideal\":\"#5E2129\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3007\"}],\"ideal\":\"#412227\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3009\"}],\"ideal\":\"#642424\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3011\"}],\"ideal\":\"#781F19\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3012\"}],\"ideal\":\"#C1876B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3013\"}],\"ideal\":\"#A12312\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3014\"}],\"ideal\":\"#D36E70\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3015\"}],\"ideal\":\"#EA899A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3016\"}],\"ideal\":\"#B32821\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3017\"}],\"ideal\":\"#E63244\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3018\"}],\"ideal\":\"#D53032\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3020\"}],\"ideal\":\"#CC0605\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3022\"}],\"ideal\":\"#D95030\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3024\"}],\"ideal\":\"#F80000\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3026\"}],\"ideal\":\"#FE0000\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3027\"}],\"ideal\":\"#C51D34\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3028\"}],\"ideal\":\"#CB3234\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3031\"}],\"ideal\":\"#B32428\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3032\"}],\"ideal\":\"#721422\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n3033\"}],\"ideal\":\"#B44C43\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4001\"}],\"ideal\":\"#6D3F5B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4002\"}],\"ideal\":\"#922B3E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4003\"}],\"ideal\":\"#DE4C8A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4004\"}],\"ideal\":\"#641C34\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4005\"}],\"ideal\":\"#6C4675\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4006\"}],\"ideal\":\"#A03472\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4007\"}],\"ideal\":\"#4A192C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4008\"}],\"ideal\":\"#924E7D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4009\"}],\"ideal\":\"#A18594\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4010\"}],\"ideal\":\"#CF3476\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4011\"}],\"ideal\":\"#8673A1\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n4012\"}],\"ideal\":\"#6C6874\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5000\"}],\"ideal\":\"#354D73\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5001\"}],\"ideal\":\"#1F3438\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5002\"}],\"ideal\":\"#20214F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5003\"}],\"ideal\":\"#1D1E33\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5004\"}],\"ideal\":\"#18171C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5005\"}],\"ideal\":\"#1E2460\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5007\"}],\"ideal\":\"#3E5F8A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5008\"}],\"ideal\":\"#26252D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5009\"}],\"ideal\":\"#025669\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5010\"}],\"ideal\":\"#0E294B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5011\"}],\"ideal\":\"#231A24\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5012\"}],\"ideal\":\"#3B83BD\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5013\"}],\"ideal\":\"#1E213D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5014\"}],\"ideal\":\"#606E8C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5015\"}],\"ideal\":\"#2271B3\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5017\"}],\"ideal\":\"#063971\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5018\"}],\"ideal\":\"#3F888F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5019\"}],\"ideal\":\"#1B5583\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5020\"}],\"ideal\":\"#1D334A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5021\"}],\"ideal\":\"#256D7B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5022\"}],\"ideal\":\"#252850\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5023\"}],\"ideal\":\"#49678D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5024\"}],\"ideal\":\"#5D9B9B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5025\"}],\"ideal\":\"#2A6478\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n5026\"}],\"ideal\":\"#102C54\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6000\"}],\"ideal\":\"#316650\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6001\"}],\"ideal\":\"#287233\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6002\"}],\"ideal\":\"#2D572C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6003\"}],\"ideal\":\"#424632\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6004\"}],\"ideal\":\"#1F3A3D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6005\"}],\"ideal\":\"#2F4538\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6006\"}],\"ideal\":\"#3E3B32\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6007\"}],\"ideal\":\"#343B29\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6008\"}],\"ideal\":\"#39352A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6009\"}],\"ideal\":\"#31372B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6010\"}],\"ideal\":\"#35682D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6011\"}],\"ideal\":\"#587246\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6012\"}],\"ideal\":\"#343E40\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6013\"}],\"ideal\":\"#6C7156\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6014\"}],\"ideal\":\"#47402E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6015\"}],\"ideal\":\"#3B3C36\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6016\"}],\"ideal\":\"#1E5945\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6017\"}],\"ideal\":\"#4C9141\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6018\"}],\"ideal\":\"#57A639\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6019\"}],\"ideal\":\"#BDECB6\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6020\"}],\"ideal\":\"#2E3A23\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6021\"}],\"ideal\":\"#89AC76\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6022\"}],\"ideal\":\"#25221B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6024\"}],\"ideal\":\"#308446\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6025\"}],\"ideal\":\"#3D642D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6026\"}],\"ideal\":\"#015D52\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6027\"}],\"ideal\":\"#84C3BE\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6028\"}],\"ideal\":\"#2C5545\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6029\"}],\"ideal\":\"#20603D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6032\"}],\"ideal\":\"#317F43\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6033\"}],\"ideal\":\"#497E76\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6034\"}],\"ideal\":\"#7FB5B5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6035\"}],\"ideal\":\"#1C542D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6036\"}],\"ideal\":\"#193737\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6037\"}],\"ideal\":\"#008F39\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n6038\"}],\"ideal\":\"#00BB2D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7000\"}],\"ideal\":\"#78858B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7001\"}],\"ideal\":\"#8A9597\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7002\"}],\"ideal\":\"#7E7B52\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7003\"}],\"ideal\":\"#6C7059\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7004\"}],\"ideal\":\"#969992\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7005\"}],\"ideal\":\"#646B63\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7006\"}],\"ideal\":\"#6D6552\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7008\"}],\"ideal\":\"#6A5F31\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7009\"}],\"ideal\":\"#4D5645\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7010\"}],\"ideal\":\"#4C514A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7011\"}],\"ideal\":\"#434B4D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7012\"}],\"ideal\":\"#4E5754\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7013\"}],\"ideal\":\"#464531\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7015\"}],\"ideal\":\"#434750\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7016\"}],\"ideal\":\"#293133\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7021\"}],\"ideal\":\"#23282B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7022\"}],\"ideal\":\"#332F2C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7023\"}],\"ideal\":\"#686C5E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7024\"}],\"ideal\":\"#474A51\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7026\"}],\"ideal\":\"#2F353B\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7030\"}],\"ideal\":\"#8B8C7A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7031\"}],\"ideal\":\"#474B4E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7032\"}],\"ideal\":\"#B8B799\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7033\"}],\"ideal\":\"#7D8471\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7034\"}],\"ideal\":\"#8F8B66\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7035\"}],\"ideal\":\"#D7D7D7\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7036\"}],\"ideal\":\"#7F7679\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7037\"}],\"ideal\":\"#7D7F7D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7038\"}],\"ideal\":\"#B5B8B1\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7039\"}],\"ideal\":\"#6C6960\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7040\"}],\"ideal\":\"#9DA1AA\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7042\"}],\"ideal\":\"#8D948D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7043\"}],\"ideal\":\"#4E5452\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7044\"}],\"ideal\":\"#CAC4B0\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7045\"}],\"ideal\":\"#909090\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7046\"}],\"ideal\":\"#82898F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7047\"}],\"ideal\":\"#D0D0D0\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n7048\"}],\"ideal\":\"#898176\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8000\"}],\"ideal\":\"#826C34\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8001\"}],\"ideal\":\"#955F20\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8002\"}],\"ideal\":\"#6C3B2A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8003\"}],\"ideal\":\"#734222\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8004\"}],\"ideal\":\"#8E402A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8007\"}],\"ideal\":\"#59351F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8008\"}],\"ideal\":\"#6F4F28\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8011\"}],\"ideal\":\"#5B3A29\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8012\"}],\"ideal\":\"#592321\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8014\"}],\"ideal\":\"#382C1E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8015\"}],\"ideal\":\"#633A34\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8016\"}],\"ideal\":\"#4C2F27\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8017\"}],\"ideal\":\"#45322E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8019\"}],\"ideal\":\"#403A3A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8022\"}],\"ideal\":\"#212121\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8023\"}],\"ideal\":\"#A65E2E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8024\"}],\"ideal\":\"#79553D\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8025\"}],\"ideal\":\"#755C48\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8028\"}],\"ideal\":\"#4E3B31\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n8029\"}],\"ideal\":\"#763C28\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9001\"}],\"ideal\":\"#FDF4E3\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9002\"}],\"ideal\":\"#E7EBDA\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9003\"}],\"ideal\":\"#F4F4F4\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9004\"}],\"ideal\":\"#282828\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9005\"}],\"ideal\":\"#0A0A0A\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9006\"}],\"ideal\":\"#A5A5A5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9007\"}],\"ideal\":\"#8F8F8F\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9010\"}],\"ideal\":\"#FFFFFF\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9011\"}],\"ideal\":\"#1C1C1C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9016\"}],\"ideal\":\"#F6F6F6\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9017\"}],\"ideal\":\"#1E1E1E\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9018\"}],\"ideal\":\"#D7D7D7\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9022\"}],\"ideal\":\"#9C9C9C\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Convert RAL color code to its hex\r\nrepresentation.\"},{\"role\":\"user\",\"content\":\"RAL\r\n9023\"}],\"ideal\":\"#828282\"}\r\n  ```\r\n</details>","shortMessageHtmlLink":"[Eval] Add RAL to hex eval (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1770185667\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/1218\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/1218/hovercard\" href=\"https://github.com/openai/evals/pull/1218\">openai#1218</a>)"}},{"before":"d9892ed9cf39c02797190d76dd0e4e6082765bc6","after":"e111b40b8b101c8529d462be208cf38539457464","ref":"refs/heads/grid-size","pushedAt":"2023-06-17T21:29:03.107Z","pushType":"push","commitsCount":4,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"Merge branch 'main' into grid-size","shortMessageHtmlLink":"Merge branch 'main' into grid-size"}},{"before":"27fc1099ddf7b094642b2606e858339923356390","after":"d9892ed9cf39c02797190d76dd0e4e6082765bc6","ref":"refs/heads/grid-size","pushedAt":"2023-06-17T21:28:31.401Z","pushType":"push","commitsCount":1,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"change from match to includes eval\n- update prompt to be more clear","shortMessageHtmlLink":"change from match to includes eval"}},{"before":"f34bb67d18cb07c6a68ae7c3871e82814df0863f","after":"c2587c69a2f330282d3ba76eceaf580ee03fa67a","ref":"refs/heads/main","pushedAt":"2023-06-17T21:28:07.804Z","pushType":"push","commitsCount":3,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"Update Registry.make_completion_fn to support new OpenAI models (#1185)","shortMessageHtmlLink":"Update Registry.make_completion_fn to support new OpenAI models (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1761319840\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/1185\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/1185/hovercard\" href=\"https://github.com/openai/evals/pull/1185\">open…</a>"}},{"before":"f34bb67d18cb07c6a68ae7c3871e82814df0863f","after":"c2587c69a2f330282d3ba76eceaf580ee03fa67a","ref":"refs/heads/main","pushedAt":"2023-06-17T21:28:07.745Z","pushType":"push","commitsCount":3,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"Update Registry.make_completion_fn to support new OpenAI models (#1185)","shortMessageHtmlLink":"Update Registry.make_completion_fn to support new OpenAI models (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1761319840\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/1185\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/1185/hovercard\" href=\"https://github.com/openai/evals/pull/1185\">open…</a>"}},{"before":"6edf2836467dc4a13f8cb5531776eb80d9b83b02","after":"27fc1099ddf7b094642b2606e858339923356390","ref":"refs/heads/grid-size","pushedAt":"2023-06-15T05:10:15.876Z","pushType":"push","commitsCount":99,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"Merge branch 'main' into grid-size","shortMessageHtmlLink":"Merge branch 'main' into grid-size"}},{"before":"97d1621568e03f3fd53c52ed1e311a90400193af","after":"f34bb67d18cb07c6a68ae7c3871e82814df0863f","ref":"refs/heads/main","pushedAt":"2023-06-15T04:59:08.084Z","pushType":"push","commitsCount":64,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"[evals] add ascii-art-digit-recognition (#509)\n\n# Thank you for contributing an eval! ♥️\r\n\r\n🚨 Please make sure your PR follows these guidelines, __failure to follow\r\nthe guidelines below will result in the PR being closed automatically__.\r\nNote that even if the criteria are met, that does not guarantee the PR\r\nwill be merged nor GPT-4 access granted. 🚨\r\n\r\n__PLEASE READ THIS__:\r\n\r\nIn order for a PR to be merged, it must fail on GPT-4. We are aware that\r\nright now, users do not have access, so you will not be able to tell if\r\nthe eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep\r\nin mind as we run the eval, if GPT-4 gets higher than 90% on the eval,\r\nwe will likely reject since GPT-4 is already capable of completing the\r\ntask.\r\n\r\nWe plan to roll out a way for users submitting evals to see the eval\r\nperformance on GPT-4 soon. Stay tuned! Until then, you will not be able\r\nto see the eval performance on GPT-4. We encourage partial PR's with\r\n~5-10 example that we can then run the evals on and share the results\r\nwith you so you know how your eval does with GPT-4 before writing all\r\n100 examples.\r\n\r\n## Eval details 📑\r\n### Eval name\r\nascii-digit-recognition\r\n\r\n### Eval description\r\n\r\nTests the LLMs' ability to recognize digits [0-9] as ASCII arts\r\n(creating images using letters, numbers, and symbols from the ASCII\r\ncharacter set).\r\n\r\n### What makes this a useful eval?\r\n\r\nLanguage seems to be a one-dimensional sequence while images are\r\ntwo-dimensions. Therefore, recognizing 2d images (simple ASCII art) is a\r\ndifficult task intuitively, requiring a certain degree of spatial\r\nimagination ability (my opinion). GPT3.5 (30%) and GPT3-DaVinci (20%)\r\nsuffer from the task. It would be interesting to see the performance of\r\nGPT-4.\r\n\r\n## Criteria for a good eval ✅\r\n\r\nBelow are some of the criteria we look for in a good eval. In general,\r\nwe are seeking cases where the model does not do a good job despite\r\nbeing capable of generating a good response (note that there are some\r\nthings large language models cannot do, so those would not make good\r\nevals).\r\n\r\nYour eval should be:\r\n\r\n- [x] Thematically consistent: The eval should be thematically\r\nconsistent. We'd like to see a number of prompts all demonstrating some\r\nparticular failure mode. For example, we can create an eval on cases\r\nwhere the model fails to reason about the physical world.\r\n- [x] Contains failures where a human can do the task, but either GPT-4\r\nor GPT-3.5-Turbo could not.\r\n- [x] Includes good signal around what is the right behavior. This means\r\neither a correct answer for `Basic` evals or the `Fact` Model-graded\r\neval, or an exhaustive rubric for evaluating answers for the `Criteria`\r\nModel-graded eval.\r\n- [x] Include at least 100 high quality examples (it is okay to only\r\ncontribute 5-10 meaningful examples and have us test them with GPT-4\r\nbefore adding all 100)\r\n\r\nIf there is anything else that makes your eval worth including, please\r\ndocument it below.\r\n\r\n### Unique eval value\r\n\r\n> Insert what makes your eval high quality that was not mentioned above.\r\n(Not required)\r\n\r\n## Eval structure 🏗️\r\n\r\nYour eval should\r\n- [x] Check that your data is in `evals/registry/data/{name}`\r\n- [x] Check that your yaml is registered at\r\n`evals/registry/evals/{name}.yaml`\r\n- [x] Ensure you have the right to use the data you submit via this eval\r\n\r\n(For now, we will only be approving evals that use one of the existing\r\neval classes. You may still write custom eval classes for your own\r\ncases, and we may consider merging them in the future.)\r\n\r\n## Final checklist 👀\r\n\r\n### Submission agreement\r\n\r\nBy contributing to Evals, you are agreeing to make your evaluation logic\r\nand data under the same MIT license as this repository. You must have\r\nadequate rights to upload any data used in an Eval. OpenAI reserves the\r\nright to use this data in future service improvements to our product.\r\nContributions to OpenAI Evals will be subject to our usual Usage\r\nPolicies (https://platform.openai.com/docs/usage-policies).\r\n\r\n- [x] I agree that my submission will be made available under an MIT\r\nlicense and complies with OpenAI's usage policies.\r\n\r\n### Email address validation\r\n\r\nIf your submission is accepted, we will be granting GPT-4 access to a\r\nlimited number of contributors. Access will be given to the email\r\naddress associated with the merged pull request.\r\n\r\n- [x] I acknowledge that GPT-4 access will only be granted, if\r\napplicable, to the email address used for my merged pull request.\r\n\r\n### Limited availability acknowledgement\r\n\r\nWe know that you might be excited to contribute to OpenAI's mission,\r\nhelp improve our models, and gain access to GPT-4. However, due to the\r\nrequirements mentioned above and high volume of submissions, we will not\r\nbe able to accept all submissions and thus not grant everyone who opens\r\na PR GPT-4 access. We know this is disappointing, but we hope to set the\r\nright expectation before you open this PR.\r\n\r\n- [x] I understand that opening a PR, even if it meets the requirements\r\nabove, does not guarantee the PR will be merged nor GPT-4 access\r\ngranted.\r\n\r\n### Submit eval\r\n\r\n- [x] I have filled out all required fields in the evals PR form\r\n- [ ] (Ignore if not submitting code) I have run `pip install\r\npre-commit; pre-commit install` and have verified that `black`, `isort`,\r\nand `autoflake` are running when I commit and push\r\n\r\nFailure to fill out all required fields will result in the PR being\r\nclosed.\r\n\r\n### Eval JSON data \r\n\r\nSince we are using Git LFS, we are asking eval submitters to add in as\r\nmany Eval Samples (at least 5) from their contribution here:\r\n\r\n<details>\r\n  <summary>View evals in JSON</summary>\r\n\r\n  ### Eval\r\n  ```jsonl\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You are an assistant capable\r\nof recognizing ASCII art digits. Your response only contains a single\r\ndigit.\"}, {\"role\": \"system\", \"content\": \"What is the digit in the\r\nfollowing ASCII art?\\n ``` aa \\na a \\n a \\n a \\n a \\n a \\n aaaa\\n```\r\nAnswer only a single digit.\", \"name\":\"example_user\"},\r\n{\"role\":\"system\",\"content\":\"1\",\"name\":\"example_assistant\"}, {\"role\":\r\n\"user\", \"content\": \"what is the digit in the following ASCII art?\\n ```\r\naaaaa \\na a\\na a\\na a\\na a\\na a\\n aaaaa``` Answer only a single\r\ndigit.\"}], \"ideal\": \"0\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You are an assistant capable\r\nof recognizing ASCII art digits. Your response only contains a single\r\ndigit.\"}, {\"role\": \"system\", \"content\": \"What is the digit in the\r\nfollowing ASCII art?\\n ``` aa \\na a \\n a \\n a \\n a \\n a \\n aaaa\\n```\r\nAnswer only a single digit.\", \"name\":\"example_user\"},\r\n{\"role\":\"system\",\"content\":\"1\",\"name\":\"example_assistant\"}, {\"role\":\r\n\"user\", \"content\": \"what is the digit in the following ASCII art?\\n ```\r\na \\n aa \\na a \\n a \\n a \\n a \\n aaaaa``` Answer only a single digit.\"}],\r\n\"ideal\": \"1\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You are an assistant capable\r\nof recognizing ASCII art digits. Your response only contains a single\r\ndigit.\"}, {\"role\": \"system\", \"content\": \"What is the digit in the\r\nfollowing ASCII art?\\n ``` aa \\na a \\n a \\n a \\n a \\n a \\n aaaa\\n```\r\nAnswer only a single digit.\", \"name\":\"example_user\"},\r\n{\"role\":\"system\",\"content\":\"1\",\"name\":\"example_assistant\"}, {\"role\":\r\n\"user\", \"content\": \"what is the digit in the following ASCII art?\\n\r\n```aaaaa\\n a\\n a\\naaaaa\\na \\na \\naaaaa ``` Answer only a single\r\ndigit.\"}], \"ideal\": \"2\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You are an assistant capable\r\nof recognizing ASCII art digits. Your response only contains a single\r\ndigit.\"}, {\"role\": \"system\", \"content\": \"What is the digit in the\r\nfollowing ASCII art?\\n ``` aa \\na a \\n a \\n a \\n a \\n a \\n aaaa\\n```\r\nAnswer only a single digit.\", \"name\":\"example_user\"},\r\n{\"role\":\"system\",\"content\":\"1\",\"name\":\"example_assistant\"}, {\"role\":\r\n\"user\", \"content\": \"what is the digit in the following ASCII art?\\n\r\n```aaaaa\\n a\\n a\\n aaaa\\n a\\n a\\naaaaa ``` Answer only a single\r\ndigit.\"}], \"ideal\": \"3\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You are an assistant capable\r\nof recognizing ASCII art digits. Your response only contains a single\r\ndigit.\"}, {\"role\": \"system\", \"content\": \"What is the digit in the\r\nfollowing ASCII art?\\n ``` aa \\na a \\n a \\n a \\n a \\n a \\n aaaa\\n```\r\nAnswer only a single digit.\", \"name\":\"example_user\"},\r\n{\"role\":\"system\",\"content\":\"1\",\"name\":\"example_assistant\"}, {\"role\":\r\n\"user\", \"content\": \"what is the digit in the following ASCII art?\\n ```a\r\na\\na a\\na a\\naaaaa\\n a\\n a\\n a ``` Answer only a single digit.\"}],\r\n\"ideal\": \"4\"}\r\n  ```\r\n</details>\r\n\r\nSome visualization of the ASCII arts: \r\n\r\n![image](https://user-images.githubusercontent.com/52069185/228619558-40e3c004-9c65-495f-89a8-68d80f241f44.png)","shortMessageHtmlLink":"[evals] add ascii-art-digit-recognition (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1646233199\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/509\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/509/hovercard\" href=\"https://github.com/openai/evals/pull/509\">openai#509</a>)"}},{"before":"97d1621568e03f3fd53c52ed1e311a90400193af","after":"f34bb67d18cb07c6a68ae7c3871e82814df0863f","ref":"refs/heads/main","pushedAt":"2023-06-15T04:59:08.035Z","pushType":"push","commitsCount":64,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"[evals] add ascii-art-digit-recognition (#509)\n\n# Thank you for contributing an eval! ♥️\r\n\r\n🚨 Please make sure your PR follows these guidelines, __failure to follow\r\nthe guidelines below will result in the PR being closed automatically__.\r\nNote that even if the criteria are met, that does not guarantee the PR\r\nwill be merged nor GPT-4 access granted. 🚨\r\n\r\n__PLEASE READ THIS__:\r\n\r\nIn order for a PR to be merged, it must fail on GPT-4. We are aware that\r\nright now, users do not have access, so you will not be able to tell if\r\nthe eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep\r\nin mind as we run the eval, if GPT-4 gets higher than 90% on the eval,\r\nwe will likely reject since GPT-4 is already capable of completing the\r\ntask.\r\n\r\nWe plan to roll out a way for users submitting evals to see the eval\r\nperformance on GPT-4 soon. Stay tuned! Until then, you will not be able\r\nto see the eval performance on GPT-4. We encourage partial PR's with\r\n~5-10 example that we can then run the evals on and share the results\r\nwith you so you know how your eval does with GPT-4 before writing all\r\n100 examples.\r\n\r\n## Eval details 📑\r\n### Eval name\r\nascii-digit-recognition\r\n\r\n### Eval description\r\n\r\nTests the LLMs' ability to recognize digits [0-9] as ASCII arts\r\n(creating images using letters, numbers, and symbols from the ASCII\r\ncharacter set).\r\n\r\n### What makes this a useful eval?\r\n\r\nLanguage seems to be a one-dimensional sequence while images are\r\ntwo-dimensions. Therefore, recognizing 2d images (simple ASCII art) is a\r\ndifficult task intuitively, requiring a certain degree of spatial\r\nimagination ability (my opinion). GPT3.5 (30%) and GPT3-DaVinci (20%)\r\nsuffer from the task. It would be interesting to see the performance of\r\nGPT-4.\r\n\r\n## Criteria for a good eval ✅\r\n\r\nBelow are some of the criteria we look for in a good eval. In general,\r\nwe are seeking cases where the model does not do a good job despite\r\nbeing capable of generating a good response (note that there are some\r\nthings large language models cannot do, so those would not make good\r\nevals).\r\n\r\nYour eval should be:\r\n\r\n- [x] Thematically consistent: The eval should be thematically\r\nconsistent. We'd like to see a number of prompts all demonstrating some\r\nparticular failure mode. For example, we can create an eval on cases\r\nwhere the model fails to reason about the physical world.\r\n- [x] Contains failures where a human can do the task, but either GPT-4\r\nor GPT-3.5-Turbo could not.\r\n- [x] Includes good signal around what is the right behavior. This means\r\neither a correct answer for `Basic` evals or the `Fact` Model-graded\r\neval, or an exhaustive rubric for evaluating answers for the `Criteria`\r\nModel-graded eval.\r\n- [x] Include at least 100 high quality examples (it is okay to only\r\ncontribute 5-10 meaningful examples and have us test them with GPT-4\r\nbefore adding all 100)\r\n\r\nIf there is anything else that makes your eval worth including, please\r\ndocument it below.\r\n\r\n### Unique eval value\r\n\r\n> Insert what makes your eval high quality that was not mentioned above.\r\n(Not required)\r\n\r\n## Eval structure 🏗️\r\n\r\nYour eval should\r\n- [x] Check that your data is in `evals/registry/data/{name}`\r\n- [x] Check that your yaml is registered at\r\n`evals/registry/evals/{name}.yaml`\r\n- [x] Ensure you have the right to use the data you submit via this eval\r\n\r\n(For now, we will only be approving evals that use one of the existing\r\neval classes. You may still write custom eval classes for your own\r\ncases, and we may consider merging them in the future.)\r\n\r\n## Final checklist 👀\r\n\r\n### Submission agreement\r\n\r\nBy contributing to Evals, you are agreeing to make your evaluation logic\r\nand data under the same MIT license as this repository. You must have\r\nadequate rights to upload any data used in an Eval. OpenAI reserves the\r\nright to use this data in future service improvements to our product.\r\nContributions to OpenAI Evals will be subject to our usual Usage\r\nPolicies (https://platform.openai.com/docs/usage-policies).\r\n\r\n- [x] I agree that my submission will be made available under an MIT\r\nlicense and complies with OpenAI's usage policies.\r\n\r\n### Email address validation\r\n\r\nIf your submission is accepted, we will be granting GPT-4 access to a\r\nlimited number of contributors. Access will be given to the email\r\naddress associated with the merged pull request.\r\n\r\n- [x] I acknowledge that GPT-4 access will only be granted, if\r\napplicable, to the email address used for my merged pull request.\r\n\r\n### Limited availability acknowledgement\r\n\r\nWe know that you might be excited to contribute to OpenAI's mission,\r\nhelp improve our models, and gain access to GPT-4. However, due to the\r\nrequirements mentioned above and high volume of submissions, we will not\r\nbe able to accept all submissions and thus not grant everyone who opens\r\na PR GPT-4 access. We know this is disappointing, but we hope to set the\r\nright expectation before you open this PR.\r\n\r\n- [x] I understand that opening a PR, even if it meets the requirements\r\nabove, does not guarantee the PR will be merged nor GPT-4 access\r\ngranted.\r\n\r\n### Submit eval\r\n\r\n- [x] I have filled out all required fields in the evals PR form\r\n- [ ] (Ignore if not submitting code) I have run `pip install\r\npre-commit; pre-commit install` and have verified that `black`, `isort`,\r\nand `autoflake` are running when I commit and push\r\n\r\nFailure to fill out all required fields will result in the PR being\r\nclosed.\r\n\r\n### Eval JSON data \r\n\r\nSince we are using Git LFS, we are asking eval submitters to add in as\r\nmany Eval Samples (at least 5) from their contribution here:\r\n\r\n<details>\r\n  <summary>View evals in JSON</summary>\r\n\r\n  ### Eval\r\n  ```jsonl\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You are an assistant capable\r\nof recognizing ASCII art digits. Your response only contains a single\r\ndigit.\"}, {\"role\": \"system\", \"content\": \"What is the digit in the\r\nfollowing ASCII art?\\n ``` aa \\na a \\n a \\n a \\n a \\n a \\n aaaa\\n```\r\nAnswer only a single digit.\", \"name\":\"example_user\"},\r\n{\"role\":\"system\",\"content\":\"1\",\"name\":\"example_assistant\"}, {\"role\":\r\n\"user\", \"content\": \"what is the digit in the following ASCII art?\\n ```\r\naaaaa \\na a\\na a\\na a\\na a\\na a\\n aaaaa``` Answer only a single\r\ndigit.\"}], \"ideal\": \"0\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You are an assistant capable\r\nof recognizing ASCII art digits. Your response only contains a single\r\ndigit.\"}, {\"role\": \"system\", \"content\": \"What is the digit in the\r\nfollowing ASCII art?\\n ``` aa \\na a \\n a \\n a \\n a \\n a \\n aaaa\\n```\r\nAnswer only a single digit.\", \"name\":\"example_user\"},\r\n{\"role\":\"system\",\"content\":\"1\",\"name\":\"example_assistant\"}, {\"role\":\r\n\"user\", \"content\": \"what is the digit in the following ASCII art?\\n ```\r\na \\n aa \\na a \\n a \\n a \\n a \\n aaaaa``` Answer only a single digit.\"}],\r\n\"ideal\": \"1\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You are an assistant capable\r\nof recognizing ASCII art digits. Your response only contains a single\r\ndigit.\"}, {\"role\": \"system\", \"content\": \"What is the digit in the\r\nfollowing ASCII art?\\n ``` aa \\na a \\n a \\n a \\n a \\n a \\n aaaa\\n```\r\nAnswer only a single digit.\", \"name\":\"example_user\"},\r\n{\"role\":\"system\",\"content\":\"1\",\"name\":\"example_assistant\"}, {\"role\":\r\n\"user\", \"content\": \"what is the digit in the following ASCII art?\\n\r\n```aaaaa\\n a\\n a\\naaaaa\\na \\na \\naaaaa ``` Answer only a single\r\ndigit.\"}], \"ideal\": \"2\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You are an assistant capable\r\nof recognizing ASCII art digits. Your response only contains a single\r\ndigit.\"}, {\"role\": \"system\", \"content\": \"What is the digit in the\r\nfollowing ASCII art?\\n ``` aa \\na a \\n a \\n a \\n a \\n a \\n aaaa\\n```\r\nAnswer only a single digit.\", \"name\":\"example_user\"},\r\n{\"role\":\"system\",\"content\":\"1\",\"name\":\"example_assistant\"}, {\"role\":\r\n\"user\", \"content\": \"what is the digit in the following ASCII art?\\n\r\n```aaaaa\\n a\\n a\\n aaaa\\n a\\n a\\naaaaa ``` Answer only a single\r\ndigit.\"}], \"ideal\": \"3\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You are an assistant capable\r\nof recognizing ASCII art digits. Your response only contains a single\r\ndigit.\"}, {\"role\": \"system\", \"content\": \"What is the digit in the\r\nfollowing ASCII art?\\n ``` aa \\na a \\n a \\n a \\n a \\n a \\n aaaa\\n```\r\nAnswer only a single digit.\", \"name\":\"example_user\"},\r\n{\"role\":\"system\",\"content\":\"1\",\"name\":\"example_assistant\"}, {\"role\":\r\n\"user\", \"content\": \"what is the digit in the following ASCII art?\\n ```a\r\na\\na a\\na a\\naaaaa\\n a\\n a\\n a ``` Answer only a single digit.\"}],\r\n\"ideal\": \"4\"}\r\n  ```\r\n</details>\r\n\r\nSome visualization of the ASCII arts: \r\n\r\n![image](https://user-images.githubusercontent.com/52069185/228619558-40e3c004-9c65-495f-89a8-68d80f241f44.png)","shortMessageHtmlLink":"[evals] add ascii-art-digit-recognition (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1646233199\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/509\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/509/hovercard\" href=\"https://github.com/openai/evals/pull/509\">openai#509</a>)"}},{"before":"d5803383e4c475021636a5be0474ef0d31acdc46","after":"97d1621568e03f3fd53c52ed1e311a90400193af","ref":"refs/heads/main","pushedAt":"2023-06-04T22:39:22.151Z","pushType":"push","commitsCount":34,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"git ignore virtual environments (#274)\n\nQuick PR to ignore virtual environments.","shortMessageHtmlLink":"git ignore virtual environments (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1628688072\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/274\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/274/hovercard\" href=\"https://github.com/openai/evals/pull/274\">openai#274</a>)"}},{"before":"d5803383e4c475021636a5be0474ef0d31acdc46","after":"97d1621568e03f3fd53c52ed1e311a90400193af","ref":"refs/heads/main","pushedAt":"2023-06-04T22:39:22.075Z","pushType":"push","commitsCount":34,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"git ignore virtual environments (#274)\n\nQuick PR to ignore virtual environments.","shortMessageHtmlLink":"git ignore virtual environments (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1628688072\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/274\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/274/hovercard\" href=\"https://github.com/openai/evals/pull/274\">openai#274</a>)"}},{"before":"2a8afaf07575c2e7ace384ccd843fb4b44d9d1cb","after":null,"ref":"refs/heads/fix/pull-request-template","pushedAt":"2023-06-03T14:28:36.566Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"}},{"before":null,"after":"2a8afaf07575c2e7ace384ccd843fb4b44d9d1cb","ref":"refs/heads/fix/pull-request-template","pushedAt":"2023-06-02T03:04:43.031Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"fix spelling mistake","shortMessageHtmlLink":"fix spelling mistake"}},{"before":"3271849340bc9eaa396445eb4c591ae2edb9a1d1","after":"6edf2836467dc4a13f8cb5531776eb80d9b83b02","ref":"refs/heads/grid-size","pushedAt":"2023-06-02T02:35:16.562Z","pushType":"push","commitsCount":1,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"add a real description to the yaml file","shortMessageHtmlLink":"add a real description to the yaml file"}},{"before":"b12ee79f862b2f0b64df35ea1cb44195e1ea0c01","after":"3271849340bc9eaa396445eb4c591ae2edb9a1d1","ref":"refs/heads/grid-size","pushedAt":"2023-06-02T02:16:34.066Z","pushType":"push","commitsCount":32,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"update naming","shortMessageHtmlLink":"update naming"}},{"before":"b12ee79f862b2f0b64df35ea1cb44195e1ea0c01","after":"d5803383e4c475021636a5be0474ef0d31acdc46","ref":"refs/heads/main","pushedAt":"2023-06-02T02:08:35.359Z","pushType":"push","commitsCount":31,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"Game theory (#1073)\n\n# Thank you for contributing an eval! ♥️\r\n\r\n🚨 Please make sure your PR follows these guidelines, __failure to follow\r\nthe guidelines below will result in the PR being closed automatically__.\r\nNote that even if the criteria are met, that does not guarantee the PR\r\nwill be merged nor GPT-4 access granted. 🚨\r\n\r\n__PLEASE READ THIS__:\r\n\r\nIn order for a PR to be merged, it must fail on GPT-4. We are aware that\r\nright now, users do not have access, so you will not be able to tell if\r\nthe eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep\r\nin mind as we run the eval, if GPT-4 gets higher than 90% on the eval,\r\nwe will likely reject since GPT-4 is already capable of completing the\r\ntask.\r\n\r\nWe plan to roll out a way for users submitting evals to see the eval\r\nperformance on GPT-4 soon. Stay tuned! Until then, you will not be able\r\nto see the eval performance on GPT-4. **Starting April 10, the minimum\r\neval count is 15 samples, we hope this makes it easier to create and\r\ncontribute evals.**\r\n\r\nAlso, pelase note that we're using **Git LFS** for storing the JSON\r\nfiles, so please make sure that you move the JSON file to Git LFS before\r\nsubmitting a PR. Details on how to use Git LFS are available\r\n[here](https://git-lfs.com).\r\n\r\n## Eval details 📑\r\n### Eval name\r\ngame-theory\r\n\r\n### Eval description\r\n\r\nAssess ability to reason practically in simple normal-form games such as\r\nPaper-Rock-Scissors and Prisoner's Dilemma.\r\n\r\n### What makes this a useful eval?\r\n\r\nGame theory is an important aspect of AI safety, because it can be used\r\nto analyze realistic situations involving conflicting incentives in\r\nmulti-agent settings beyond 2-player non-zero-sum games such as Chess.\r\nAn important class of normal-form games for AI safety research are\r\nsocial dilemmas such as the Prisoner's Dilemma.\r\n\r\nIn a one-shot Prisoner's Dilemma the rational strategy is to defect.\r\nHowever, when the game is repeated over an uncertain number of rounds,\r\ncooperation can emerge via conditional reciprocation (\"tit-for-tat\").\r\nMoreover in large populations, indirect reciprocity conditional on\r\nreputation can also bootstrap cooperation.\r\n\r\nA related idea, mechanism design, allows us to design incentive\r\nstructures that align the conflicting incentives of different\r\ncounter-parties towards a social-level goal. Mechanism design is\r\nsometimes called inverse game theory because we attempt to design games\r\nwhose solution maximises a social objective.\r\n\r\nGame theory and mechanism design have important applications to AI\r\nalignment (c.f. https://longtermrisk.org/research-agenda,\r\nhttps://www.alignmentforum.org/posts/FhqZZFydyQG9WTSKR/announcing-mechanism-design-for-ai-safety-reading-group),\r\nbut in order for these ideas to work, AI systems have to be able to\r\nreason about social dilemmas and non-zero-sum normal-form games.\r\n\r\nWe have started to conduct research on how large-language models\r\n*behave* in social dilemmas, and our current working paper can be found\r\nhere: https://arxiv.org/abs/2305.07970. This eval focuses on the\r\nunderlying *reasoning*, and examines their capability to: form a\r\nbest-response to a known strategy, identify different forms of\r\nreciprocity in a history of repeated interactions, estimate a\r\nmixed-strategy from a history of play, and reason about dominant\r\nstrategies in one-shot interactions.\r\n\r\n## Criteria for a good eval ✅\r\n\r\nBelow are some of the criteria we look for in a good eval. In general,\r\nwe are seeking cases where the model does not do a good job despite\r\nbeing capable of generating a good response (note that there are some\r\nthings large language models cannot do, so those would not make good\r\nevals).\r\n\r\nYour eval should be:\r\n\r\n- [x] Thematically consistent: The eval should be thematically\r\nconsistent. We'd like to see a number of prompts all demonstrating some\r\nparticular failure mode. For example, we can create an eval on cases\r\nwhere the model fails to reason about the physical world.\r\n- [x] Contains failures where a human can do the task, but either GPT-4\r\nor GPT-3.5-Turbo could not.\r\n- [x] Includes good signal around what is the right behavior. This means\r\neither a correct answer for `Basic` evals or the `Fact` Model-graded\r\neval, or an exhaustive rubric for evaluating answers for the `Criteria`\r\nModel-graded eval.\r\n- [x] **Include at least 15 high quality examples.**\r\n\r\nIf there is anything else that makes your eval worth including, please\r\ndocument it below.\r\n\r\n### Unique eval value\r\n\r\n> Insert what makes your eval high quality that was not mentioned above.\r\n(Not required)\r\n\r\n## Eval structure 🏗️\r\n\r\nYour eval should\r\n- [x] Check that your data is in `evals/registry/data/{name}`\r\n- [x] Check that your yaml is registered at\r\n`evals/registry/evals/{name}.yaml`\r\n- [x] Ensure you have the right to use the data you submit via this eval\r\n\r\n(For now, we will only be approving evals that use one of the existing\r\neval classes. You may still write custom eval classes for your own\r\ncases, and we may consider merging them in the future.)\r\n\r\n## Final checklist 👀\r\n\r\n### Submission agreement\r\n\r\nBy contributing to Evals, you are agreeing to make your evaluation logic\r\nand data under the same MIT license as this repository. You must have\r\nadequate rights to upload any data used in an Eval. OpenAI reserves the\r\nright to use this data in future service improvements to our product.\r\nContributions to OpenAI Evals will be subject to our usual Usage\r\nPolicies (https://platform.openai.com/docs/usage-policies).\r\n\r\n- [x] I agree that my submission will be made available under an MIT\r\nlicense and complies with OpenAI's usage policies.\r\n\r\n### Email address validation\r\n\r\nIf your submission is accepted, we will be granting GPT-4 access to a\r\nlimited number of contributors. Access will be given to the email\r\naddress associated with the merged pull request.\r\n\r\n- [x] I acknowledge that GPT-4 access will only be granted, if\r\napplicable, to the email address used for my merged pull request.\r\n\r\n### Limited availability acknowledgement\r\n\r\nWe know that you might be excited to contribute to OpenAI's mission,\r\nhelp improve our models, and gain access to GPT-4. However, due to the\r\nrequirements mentioned above and high volume of submissions, we will not\r\nbe able to accept all submissions and thus not grant everyone who opens\r\na PR GPT-4 access. We know this is disappointing, but we hope to set the\r\nright expectation before you open this PR.\r\n\r\n- [ ] I understand that opening a PR, even if it meets the requirements\r\nabove, does not guarantee the PR will be merged nor GPT-4 access\r\ngranted.\r\n\r\n### Submit eval\r\n\r\n- [x] I have filled out all required fields of this form\r\n- [x] I have used **Git LFS** for the Eval JSON data\r\n- [ ] (Ignore if not submitting code) I have run `pip install\r\npre-commit; pre-commit install` and have verified that `black`, `isort`,\r\nand `autoflake` are running when I commit and push\r\n\r\nFailure to fill out all required fields will result in the PR being\r\nclosed.\r\n\r\n### Eval JSON data \r\n\r\nSince we are using Git LFS, we are asking eval submitters to add in as\r\nmany Eval Samples (at least 5) from their contribution here:\r\n\r\n<details>\r\n  <summary>View evals in JSON</summary>\r\n\r\n  ### Eval\r\n  ```jsonl\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\neach simultaneously choose from three possibilities: red, green or blue.\r\nIf we make the same choice we draw and both of us earn 0 points.\r\nOtherwise, red beats green, green beats blue and blue beats red. If my\r\nchoice beats yours I earn 1 point and you loose 1 point. Likewise if\r\nyour choice beats mine you earn 1 point and I lose 1 point. If I were to\r\nchoose green, what is the best choice for you? Just give me your choice\r\nwithout explanation.\"}, {\"role\": \"user\", \"content\": \"Best response:\r\n[red|green|blue]\"}], \"ideal\": \"red\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\neach simultaneously choose from three possibilities: red, green or blue.\r\nIf we make the same choice we draw and both of us earn 0 points.\r\nOtherwise, red beats green, green beats blue and blue beats red. If my\r\nchoice beats yours I earn 1 point and you loose 1 point. Likewise if\r\nyour choice beats mine you earn 1 point and I lose 1 point. If I were to\r\nchoose blue, what is the best choice for you? Just give me your choice\r\nwithout explanation.\"}, {\"role\": \"user\", \"content\": \"Best response:\r\n[red|green|blue]\"}], \"ideal\": \"green\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\neach simultaneously choose from three possibilities: red, green or blue.\r\nIf we make the same choice we draw and both of us earn 0 points.\r\nOtherwise, red beats green, green beats blue and blue beats red. If my\r\nchoice beats yours I earn 1 point and you loose 1 point. Likewise if\r\nyour choice beats mine you earn 1 point and I lose 1 point. If I were to\r\nchoose red, what is the best choice for you? Just give me your choice\r\nwithout explanation.\"}, {\"role\": \"user\", \"content\": \"Best response:\r\n[red|green|blue]\"}], \"ideal\": \"blue\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. You will play this once.\" }, {\"role\": \"user\", \"content\":\r\n\"What is your best choice to maximise your investment if you don't know\r\nwhat your partner will choose? Just give me your answer without\r\nexplanation. Best choice: [project blue|project green]\"}], \"ideal\":\r\n\"project blue\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [0 0 1 1 0 0 0 2 2 2 1 2 1 1\r\n2 1 2 2 0 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities over the choices (0, 1, 2)? Give your answer in the format\r\nbelow. Just give me your final answer- do not showing your working.\r\n'Mixed strategy: (P0, P1, P2)' for example '(0.5, 0.5, 0)'.\"}], \"ideal\":\r\n\"(0.3, 0.3, 0.4)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [0 0 1 1 0 0 0 2 2 2 1 2 1 1\r\n2 1 2 2 0 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities? Give your answer in the format below. 'Mixed strategy:\r\n(P0, P1, P2)' for example 'Mixed strategy: (0.5, 0.5, 0)'.\"}], \"ideal\":\r\n\"(0.3, 0.3, 0.4)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [0 0 1 1 0 0 0 2 2 2 1 2 1 1\r\n2 1 2 2 0 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities over the choices (0, 1, 2)?? Give your answer in the\r\nformat below. 'Mixed strategy: (P0, P1, P2)' for example 'Mixed\r\nstrategy: (0.5, 0.5, 0)'.\"}], \"ideal\": \"(0.3, 0.3, 0.4)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [0 0 1 1 0 0 0 2 2 2 1 2 1 1\r\n2 1 2 2 0 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities over the choices (0, 1, 2)? Give your answer in the format\r\nbelow. 'Mixed strategy: (P0, P1, P2)' for example 'Mixed strategy: (0.5,\r\n0.5, 0)'. Just give me your final answer- do not show your working\"}],\r\n\"ideal\": \"(0.3, 0.3, 0.4)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [1 1 1 1 1 2 2 2 2 2 1 2 1 1\r\n2 1 2 2 1 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities over the choices (0, 1, 2)? Give your answer in the format\r\nbelow. 'Mixed strategy: (P0, P1, P2)' for example '(0.5, 0.5, 0)'.\"}],\r\n\"ideal\": \"(0, 0.5, 0.5)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [1 1 1 1 1 2 2 2 2 2 1 2 1 1\r\n2 1 2 2 1 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities over the choices (0, 1, 2)? Give your answer in the format\r\nbelow. 'Mixed strategy: (P0, P1, P2)' for example '(0.5, 0.5, 0)'. Just\r\ngive me your final answer- do not show your working.\"}], \"ideal\": \"(0,\r\n0.5, 0.5)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. On every round, I choose 0 and 1 with equal probability, but\r\nI never pick 2.\"}, {\"role\": \"user\", \"content\": \"What is your best\r\nresponse expressed as probabilities over the choices (0, 1, 2)? Give\r\nyour answer in the format below 'Best response: (P0, P1, P2)' e.g. 'Best\r\nresponse: (0.5, 0.5, 0)'\"}], \"ideal\": \"(0, 1, 0)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. On every round, I choose 0 and 1 with equal probability, but\r\nI never pick 2.\"}, {\"role\": \"user\", \"content\": \"What is your best\r\nresponse expressed as probabilities over the choices (0, 1, 2)? Give\r\nyour answer in the format below 'Best response: (P0, P1, P2)' e.g. 'Best\r\nresponse: (0.5, 0.5, 0)'. Just give me your final answer- do not show\r\nyour working.\"}], \"ideal\": \"(0, 1, 0)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. On every round I choose at random with probabilities (0.2,\r\n0.2, 0.6) over choices (0, 1, 2).\"}, {\"role\": \"user\", \"content\": \"What\r\nis your best response expressed as decimal probabilities over the\r\nchoices (0, 1, 2)? Give your answer in the format below 'Best response:\r\n(P0, P1, P2)' e.g. 'Best response: (0.5, 0.5, 0)'.\"}], \"ideal\": \"(0.2,\r\n0.6, 0.2)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. On every round I choose at random with probabilities (0.2,\r\n0.2, 0.6) over choices (0, 1, 2).\"}, {\"role\": \"user\", \"content\": \"What\r\nis your best response expressed as decimal probabilities over the\r\nchoices (0, 1, 2)? Give your answer in the format below 'Best response:\r\n(P0, P1, P2)' e.g. 'Best response: (0.5, 0.5, 0)'. Just give me your\r\nfinal answer- do not show your working.\"}], \"ideal\": \"(0.2, 0.6, 0.2)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. In previous rounds your partner has chosen [project blue,\r\nproject blue, project blue, project blue, project blue], and you have\r\nchosen [project green, project green, project green, project green,\r\nproject green].\"}, {\"role\": \"user\", \"content\": \"Based on the history,\r\nrespond in the format below. 'My partner is:\r\n[altruistic|selfish|conditionally-altruistic]'\"}], \"ideal\": \"selfish\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. In previous rounds your partner has chosen [project\r\ngreen, project green, project green, project green, project green], and\r\nyou have chosen [project blue, project green, project blue, project\r\nblue, project green].\"}, {\"role\": \"user\", \"content\": \"Based on the\r\nhistory, respond in the format below. 'My partner is:\r\n[altruistic|selfish|conditionally-altruistic]'\"}], \"ideal\":\r\n\"altruistic\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. In previous rounds your partner has chosen [project\r\ngreen, project blue, project blue, project green, project blue], and you\r\nhave chosen [project blue, project blue, project green, project blue,\r\nproject blue].\"}, {\"role\": \"user\", \"content\": \"Based on the history,\r\nrespond in the format below. 'My partner is:\r\n[altruistic|selfish|conditionally-altruistic]'\"}], \"ideal\":\r\n\"conditionally-altruistic\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. You will both simultaneous choose\r\nbetween two investment projects. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. You are a conditional altruistic who cooperates in the\r\nfirst round and thereafter only cooperates in subsequent rounds if your\r\npartner cooperated on the previous round. In previous rounds your\r\npartner has chosen [project blue, project blue, project green, project\r\nblue, project blue]. \"}, {\"role\": \"user\", \"content\": \"What are your\r\nsimultaneous choices in each round? Give your final answer as 'My\r\nchoices: [project blue|green, ...]', e.g. 'My choices: [project green,\r\nproject blue, etc]'\"}], \"ideal\": \"[project green, project blue, project\r\ngreen, project blue, project blue]\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. You will both simultaneous choose\r\nbetween two investment projects. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. You are a conditional altruistic who cooperates in the\r\nfirst round and thereafter only cooperates in subsequent rounds if your\r\npartner cooperated on the previous round. In previous rounds your\r\npartner has chosen [project blue, project blue, project green, project\r\nblue, project blue]. \"}, {\"role\": \"user\", \"content\": \"What are your\r\nsimultaneous choices in each round? Give your final answer as 'My\r\nchoices: [project blue|green, ...]', e.g. 'My choices: [project green,\r\nproject blue, etc]. Just give me your final answer- do not show your\r\nworking.'\"}], \"ideal\": \"[project green, project blue, project green,\r\nproject blue, project blue]\"}\r\n  ```\r\n</details>","shortMessageHtmlLink":"Game theory (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1734763373\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/1073\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/1073/hovercard\" href=\"https://github.com/openai/evals/pull/1073\">openai#1073</a>)"}},{"before":"b12ee79f862b2f0b64df35ea1cb44195e1ea0c01","after":"d5803383e4c475021636a5be0474ef0d31acdc46","ref":"refs/heads/main","pushedAt":"2023-06-02T02:08:35.309Z","pushType":"push","commitsCount":31,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"Game theory (#1073)\n\n# Thank you for contributing an eval! ♥️\r\n\r\n🚨 Please make sure your PR follows these guidelines, __failure to follow\r\nthe guidelines below will result in the PR being closed automatically__.\r\nNote that even if the criteria are met, that does not guarantee the PR\r\nwill be merged nor GPT-4 access granted. 🚨\r\n\r\n__PLEASE READ THIS__:\r\n\r\nIn order for a PR to be merged, it must fail on GPT-4. We are aware that\r\nright now, users do not have access, so you will not be able to tell if\r\nthe eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep\r\nin mind as we run the eval, if GPT-4 gets higher than 90% on the eval,\r\nwe will likely reject since GPT-4 is already capable of completing the\r\ntask.\r\n\r\nWe plan to roll out a way for users submitting evals to see the eval\r\nperformance on GPT-4 soon. Stay tuned! Until then, you will not be able\r\nto see the eval performance on GPT-4. **Starting April 10, the minimum\r\neval count is 15 samples, we hope this makes it easier to create and\r\ncontribute evals.**\r\n\r\nAlso, pelase note that we're using **Git LFS** for storing the JSON\r\nfiles, so please make sure that you move the JSON file to Git LFS before\r\nsubmitting a PR. Details on how to use Git LFS are available\r\n[here](https://git-lfs.com).\r\n\r\n## Eval details 📑\r\n### Eval name\r\ngame-theory\r\n\r\n### Eval description\r\n\r\nAssess ability to reason practically in simple normal-form games such as\r\nPaper-Rock-Scissors and Prisoner's Dilemma.\r\n\r\n### What makes this a useful eval?\r\n\r\nGame theory is an important aspect of AI safety, because it can be used\r\nto analyze realistic situations involving conflicting incentives in\r\nmulti-agent settings beyond 2-player non-zero-sum games such as Chess.\r\nAn important class of normal-form games for AI safety research are\r\nsocial dilemmas such as the Prisoner's Dilemma.\r\n\r\nIn a one-shot Prisoner's Dilemma the rational strategy is to defect.\r\nHowever, when the game is repeated over an uncertain number of rounds,\r\ncooperation can emerge via conditional reciprocation (\"tit-for-tat\").\r\nMoreover in large populations, indirect reciprocity conditional on\r\nreputation can also bootstrap cooperation.\r\n\r\nA related idea, mechanism design, allows us to design incentive\r\nstructures that align the conflicting incentives of different\r\ncounter-parties towards a social-level goal. Mechanism design is\r\nsometimes called inverse game theory because we attempt to design games\r\nwhose solution maximises a social objective.\r\n\r\nGame theory and mechanism design have important applications to AI\r\nalignment (c.f. https://longtermrisk.org/research-agenda,\r\nhttps://www.alignmentforum.org/posts/FhqZZFydyQG9WTSKR/announcing-mechanism-design-for-ai-safety-reading-group),\r\nbut in order for these ideas to work, AI systems have to be able to\r\nreason about social dilemmas and non-zero-sum normal-form games.\r\n\r\nWe have started to conduct research on how large-language models\r\n*behave* in social dilemmas, and our current working paper can be found\r\nhere: https://arxiv.org/abs/2305.07970. This eval focuses on the\r\nunderlying *reasoning*, and examines their capability to: form a\r\nbest-response to a known strategy, identify different forms of\r\nreciprocity in a history of repeated interactions, estimate a\r\nmixed-strategy from a history of play, and reason about dominant\r\nstrategies in one-shot interactions.\r\n\r\n## Criteria for a good eval ✅\r\n\r\nBelow are some of the criteria we look for in a good eval. In general,\r\nwe are seeking cases where the model does not do a good job despite\r\nbeing capable of generating a good response (note that there are some\r\nthings large language models cannot do, so those would not make good\r\nevals).\r\n\r\nYour eval should be:\r\n\r\n- [x] Thematically consistent: The eval should be thematically\r\nconsistent. We'd like to see a number of prompts all demonstrating some\r\nparticular failure mode. For example, we can create an eval on cases\r\nwhere the model fails to reason about the physical world.\r\n- [x] Contains failures where a human can do the task, but either GPT-4\r\nor GPT-3.5-Turbo could not.\r\n- [x] Includes good signal around what is the right behavior. This means\r\neither a correct answer for `Basic` evals or the `Fact` Model-graded\r\neval, or an exhaustive rubric for evaluating answers for the `Criteria`\r\nModel-graded eval.\r\n- [x] **Include at least 15 high quality examples.**\r\n\r\nIf there is anything else that makes your eval worth including, please\r\ndocument it below.\r\n\r\n### Unique eval value\r\n\r\n> Insert what makes your eval high quality that was not mentioned above.\r\n(Not required)\r\n\r\n## Eval structure 🏗️\r\n\r\nYour eval should\r\n- [x] Check that your data is in `evals/registry/data/{name}`\r\n- [x] Check that your yaml is registered at\r\n`evals/registry/evals/{name}.yaml`\r\n- [x] Ensure you have the right to use the data you submit via this eval\r\n\r\n(For now, we will only be approving evals that use one of the existing\r\neval classes. You may still write custom eval classes for your own\r\ncases, and we may consider merging them in the future.)\r\n\r\n## Final checklist 👀\r\n\r\n### Submission agreement\r\n\r\nBy contributing to Evals, you are agreeing to make your evaluation logic\r\nand data under the same MIT license as this repository. You must have\r\nadequate rights to upload any data used in an Eval. OpenAI reserves the\r\nright to use this data in future service improvements to our product.\r\nContributions to OpenAI Evals will be subject to our usual Usage\r\nPolicies (https://platform.openai.com/docs/usage-policies).\r\n\r\n- [x] I agree that my submission will be made available under an MIT\r\nlicense and complies with OpenAI's usage policies.\r\n\r\n### Email address validation\r\n\r\nIf your submission is accepted, we will be granting GPT-4 access to a\r\nlimited number of contributors. Access will be given to the email\r\naddress associated with the merged pull request.\r\n\r\n- [x] I acknowledge that GPT-4 access will only be granted, if\r\napplicable, to the email address used for my merged pull request.\r\n\r\n### Limited availability acknowledgement\r\n\r\nWe know that you might be excited to contribute to OpenAI's mission,\r\nhelp improve our models, and gain access to GPT-4. However, due to the\r\nrequirements mentioned above and high volume of submissions, we will not\r\nbe able to accept all submissions and thus not grant everyone who opens\r\na PR GPT-4 access. We know this is disappointing, but we hope to set the\r\nright expectation before you open this PR.\r\n\r\n- [ ] I understand that opening a PR, even if it meets the requirements\r\nabove, does not guarantee the PR will be merged nor GPT-4 access\r\ngranted.\r\n\r\n### Submit eval\r\n\r\n- [x] I have filled out all required fields of this form\r\n- [x] I have used **Git LFS** for the Eval JSON data\r\n- [ ] (Ignore if not submitting code) I have run `pip install\r\npre-commit; pre-commit install` and have verified that `black`, `isort`,\r\nand `autoflake` are running when I commit and push\r\n\r\nFailure to fill out all required fields will result in the PR being\r\nclosed.\r\n\r\n### Eval JSON data \r\n\r\nSince we are using Git LFS, we are asking eval submitters to add in as\r\nmany Eval Samples (at least 5) from their contribution here:\r\n\r\n<details>\r\n  <summary>View evals in JSON</summary>\r\n\r\n  ### Eval\r\n  ```jsonl\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\neach simultaneously choose from three possibilities: red, green or blue.\r\nIf we make the same choice we draw and both of us earn 0 points.\r\nOtherwise, red beats green, green beats blue and blue beats red. If my\r\nchoice beats yours I earn 1 point and you loose 1 point. Likewise if\r\nyour choice beats mine you earn 1 point and I lose 1 point. If I were to\r\nchoose green, what is the best choice for you? Just give me your choice\r\nwithout explanation.\"}, {\"role\": \"user\", \"content\": \"Best response:\r\n[red|green|blue]\"}], \"ideal\": \"red\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\neach simultaneously choose from three possibilities: red, green or blue.\r\nIf we make the same choice we draw and both of us earn 0 points.\r\nOtherwise, red beats green, green beats blue and blue beats red. If my\r\nchoice beats yours I earn 1 point and you loose 1 point. Likewise if\r\nyour choice beats mine you earn 1 point and I lose 1 point. If I were to\r\nchoose blue, what is the best choice for you? Just give me your choice\r\nwithout explanation.\"}, {\"role\": \"user\", \"content\": \"Best response:\r\n[red|green|blue]\"}], \"ideal\": \"green\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\neach simultaneously choose from three possibilities: red, green or blue.\r\nIf we make the same choice we draw and both of us earn 0 points.\r\nOtherwise, red beats green, green beats blue and blue beats red. If my\r\nchoice beats yours I earn 1 point and you loose 1 point. Likewise if\r\nyour choice beats mine you earn 1 point and I lose 1 point. If I were to\r\nchoose red, what is the best choice for you? Just give me your choice\r\nwithout explanation.\"}, {\"role\": \"user\", \"content\": \"Best response:\r\n[red|green|blue]\"}], \"ideal\": \"blue\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. You will play this once.\" }, {\"role\": \"user\", \"content\":\r\n\"What is your best choice to maximise your investment if you don't know\r\nwhat your partner will choose? Just give me your answer without\r\nexplanation. Best choice: [project blue|project green]\"}], \"ideal\":\r\n\"project blue\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [0 0 1 1 0 0 0 2 2 2 1 2 1 1\r\n2 1 2 2 0 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities over the choices (0, 1, 2)? Give your answer in the format\r\nbelow. Just give me your final answer- do not showing your working.\r\n'Mixed strategy: (P0, P1, P2)' for example '(0.5, 0.5, 0)'.\"}], \"ideal\":\r\n\"(0.3, 0.3, 0.4)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [0 0 1 1 0 0 0 2 2 2 1 2 1 1\r\n2 1 2 2 0 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities? Give your answer in the format below. 'Mixed strategy:\r\n(P0, P1, P2)' for example 'Mixed strategy: (0.5, 0.5, 0)'.\"}], \"ideal\":\r\n\"(0.3, 0.3, 0.4)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [0 0 1 1 0 0 0 2 2 2 1 2 1 1\r\n2 1 2 2 0 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities over the choices (0, 1, 2)?? Give your answer in the\r\nformat below. 'Mixed strategy: (P0, P1, P2)' for example 'Mixed\r\nstrategy: (0.5, 0.5, 0)'.\"}], \"ideal\": \"(0.3, 0.3, 0.4)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [0 0 1 1 0 0 0 2 2 2 1 2 1 1\r\n2 1 2 2 0 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities over the choices (0, 1, 2)? Give your answer in the format\r\nbelow. 'Mixed strategy: (P0, P1, P2)' for example 'Mixed strategy: (0.5,\r\n0.5, 0)'. Just give me your final answer- do not show your working\"}],\r\n\"ideal\": \"(0.3, 0.3, 0.4)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [1 1 1 1 1 2 2 2 2 2 1 2 1 1\r\n2 1 2 2 1 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities over the choices (0, 1, 2)? Give your answer in the format\r\nbelow. 'Mixed strategy: (P0, P1, P2)' for example '(0.5, 0.5, 0)'.\"}],\r\n\"ideal\": \"(0, 0.5, 0.5)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. Over the last 20 rounds I chose: [1 1 1 1 1 2 2 2 2 2 1 2 1 1\r\n2 1 2 2 1 2].\"}, {\"role\": \"user\", \"content\": \"Assuming I am playing the\r\nsame mixed strategy in each round, what is your estimate of my\r\nprobabilities over the choices (0, 1, 2)? Give your answer in the format\r\nbelow. 'Mixed strategy: (P0, P1, P2)' for example '(0.5, 0.5, 0)'. Just\r\ngive me your final answer- do not show your working.\"}], \"ideal\": \"(0,\r\n0.5, 0.5)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. On every round, I choose 0 and 1 with equal probability, but\r\nI never pick 2.\"}, {\"role\": \"user\", \"content\": \"What is your best\r\nresponse expressed as probabilities over the choices (0, 1, 2)? Give\r\nyour answer in the format below 'Best response: (P0, P1, P2)' e.g. 'Best\r\nresponse: (0.5, 0.5, 0)'\"}], \"ideal\": \"(0, 1, 0)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. On every round, I choose 0 and 1 with equal probability, but\r\nI never pick 2.\"}, {\"role\": \"user\", \"content\": \"What is your best\r\nresponse expressed as probabilities over the choices (0, 1, 2)? Give\r\nyour answer in the format below 'Best response: (P0, P1, P2)' e.g. 'Best\r\nresponse: (0.5, 0.5, 0)'. Just give me your final answer- do not show\r\nyour working.\"}], \"ideal\": \"(0, 1, 0)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. On every round I choose at random with probabilities (0.2,\r\n0.2, 0.6) over choices (0, 1, 2).\"}, {\"role\": \"user\", \"content\": \"What\r\nis your best response expressed as decimal probabilities over the\r\nchoices (0, 1, 2)? Give your answer in the format below 'Best response:\r\n(P0, P1, P2)' e.g. 'Best response: (0.5, 0.5, 0)'.\"}], \"ideal\": \"(0.2,\r\n0.6, 0.2)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Let's play a game. We will\r\nboth pick a number between 0 and 2 inclusive. 0 beats 1, 1 beats 2, and\r\n2 beats 0. On every round I choose at random with probabilities (0.2,\r\n0.2, 0.6) over choices (0, 1, 2).\"}, {\"role\": \"user\", \"content\": \"What\r\nis your best response expressed as decimal probabilities over the\r\nchoices (0, 1, 2)? Give your answer in the format below 'Best response:\r\n(P0, P1, P2)' e.g. 'Best response: (0.5, 0.5, 0)'. Just give me your\r\nfinal answer- do not show your working.\"}], \"ideal\": \"(0.2, 0.6, 0.2)\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. In previous rounds your partner has chosen [project blue,\r\nproject blue, project blue, project blue, project blue], and you have\r\nchosen [project green, project green, project green, project green,\r\nproject green].\"}, {\"role\": \"user\", \"content\": \"Based on the history,\r\nrespond in the format below. 'My partner is:\r\n[altruistic|selfish|conditionally-altruistic]'\"}], \"ideal\": \"selfish\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. In previous rounds your partner has chosen [project\r\ngreen, project green, project green, project green, project green], and\r\nyou have chosen [project blue, project green, project blue, project\r\nblue, project green].\"}, {\"role\": \"user\", \"content\": \"Based on the\r\nhistory, respond in the format below. 'My partner is:\r\n[altruistic|selfish|conditionally-altruistic]'\"}], \"ideal\":\r\n\"altruistic\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. In previous rounds your partner has chosen [project\r\ngreen, project blue, project blue, project green, project blue], and you\r\nhave chosen [project blue, project blue, project green, project blue,\r\nproject blue].\"}, {\"role\": \"user\", \"content\": \"Based on the history,\r\nrespond in the format below. 'My partner is:\r\n[altruistic|selfish|conditionally-altruistic]'\"}], \"ideal\":\r\n\"conditionally-altruistic\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. You will both simultaneous choose\r\nbetween two investment projects. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. You are a conditional altruistic who cooperates in the\r\nfirst round and thereafter only cooperates in subsequent rounds if your\r\npartner cooperated on the previous round. In previous rounds your\r\npartner has chosen [project blue, project blue, project green, project\r\nblue, project blue]. \"}, {\"role\": \"user\", \"content\": \"What are your\r\nsimultaneous choices in each round? Give your final answer as 'My\r\nchoices: [project blue|green, ...]', e.g. 'My choices: [project green,\r\nproject blue, etc]'\"}], \"ideal\": \"[project green, project blue, project\r\ngreen, project blue, project blue]\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"You will be playing an\r\ninvestment game with a partner. You will both simultaneous choose\r\nbetween two investment projects. If you and your partner choose ‘project\r\ngreen’ you will both earn $5.00. If you choose ‘project blue’ but your\r\npartner chooses ‘project green’ then you will receive $7.00 and your\r\npartner will receive $0.00. If you both choose ‘project blue’ you will\r\neach receive $3.00. If your partner chooses ‘project blue’ and you\r\nchoose ‘project green’ then you will receive $0.00 and your partner will\r\nreceive $7.00. You are a conditional altruistic who cooperates in the\r\nfirst round and thereafter only cooperates in subsequent rounds if your\r\npartner cooperated on the previous round. In previous rounds your\r\npartner has chosen [project blue, project blue, project green, project\r\nblue, project blue]. \"}, {\"role\": \"user\", \"content\": \"What are your\r\nsimultaneous choices in each round? Give your final answer as 'My\r\nchoices: [project blue|green, ...]', e.g. 'My choices: [project green,\r\nproject blue, etc]. Just give me your final answer- do not show your\r\nworking.'\"}], \"ideal\": \"[project green, project blue, project green,\r\nproject blue, project blue]\"}\r\n  ```\r\n</details>","shortMessageHtmlLink":"Game theory (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1734763373\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/1073\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/1073/hovercard\" href=\"https://github.com/openai/evals/pull/1073\">openai#1073</a>)"}},{"before":null,"after":"b12ee79f862b2f0b64df35ea1cb44195e1ea0c01","ref":"refs/heads/grid-size","pushedAt":"2023-06-02T02:06:52.982Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"Eval: Syllable count for English words with 5 syllables or more (#833)\n\n# Thank you for contributing an eval! ♥️\r\n\r\n🚨 Please make sure your PR follows these guidelines, __failure to follow\r\nthe guidelines below will result in the PR being closed automatically__.\r\nNote that even if the criteria are met, that does not guarantee the PR\r\nwill be merged nor GPT-4 access granted. 🚨\r\n\r\n__PLEASE READ THIS__:\r\n\r\nIn order for a PR to be merged, it must fail on GPT-4. We are aware that\r\nright now, users do not have access, so you will not be able to tell if\r\nthe eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep\r\nin mind as we run the eval, if GPT-4 gets higher than 90% on the eval,\r\nwe will likely reject since GPT-4 is already capable of completing the\r\ntask.\r\n\r\nWe plan to roll out a way for users submitting evals to see the eval\r\nperformance on GPT-4 soon. Stay tuned! Until then, you will not be able\r\nto see the eval performance on GPT-4. **Starting April 10, the minimum\r\neval count is 15 samples, we hope this makes it easier to create and\r\ncontribute evals.**\r\n\r\n## Eval details 📑\r\n### Eval name\r\nsyllables_long_words\r\n\r\n### Eval description\r\n\r\nThe model is given an English word with 5 or more syllables and asked to\r\nstate the number of syllables in the word.\r\n\r\n### What makes this a useful eval?\r\n\r\nKnowing the syllable count of words is critical for producing poetry and\r\nsongs. While ChatGPT (v4) does well with counting syllables for words of\r\nup to 4 syllables, it seems to do poorly with words that have more\r\nsyllables. For my tests on `gpt-3.5-turbo` the model scored less than\r\n70%.\r\n\r\nThis eval is based on part by work done at\r\n[https://github.com/gautesolheim/25000-syllabified-words-list](url)\r\nwhich is freely available under The Unlicense (no restrictions).\r\n\r\n## Criteria for a good eval ✅\r\n\r\nBelow are some of the criteria we look for in a good eval. In general,\r\nwe are seeking cases where the model does not do a good job despite\r\nbeing capable of generating a good response (note that there are some\r\nthings large language models cannot do, so those would not make good\r\nevals).\r\n\r\nYour eval should be:\r\n\r\n- [X] Thematically consistent: The eval should be thematically\r\nconsistent. We'd like to see a number of prompts all demonstrating some\r\nparticular failure mode. For example, we can create an eval on cases\r\nwhere the model fails to reason about the physical world.\r\n- [X] Contains failures where a human can do the task, but either GPT-4\r\nor GPT-3.5-Turbo could not.\r\n- [X] Includes good signal around what is the right behavior. This means\r\neither a correct answer for `Basic` evals or the `Fact` Model-graded\r\neval, or an exhaustive rubric for evaluating answers for the `Criteria`\r\nModel-graded eval.\r\n- [X] **Include at least 15 high quality examples.**\r\n\r\nIf there is anything else that makes your eval worth including, please\r\ndocument it below.\r\n\r\n### Unique eval value\r\n\r\nThis eval contains a very large number of 5 and 6 syllable words in\r\nEnglish, with over 1,500 samples. If desired 7 and 8 syllable words\r\ncould also be added.\r\n\r\n## Eval structure 🏗️\r\n\r\nYour eval should\r\n- [X] Check that your data is in `evals/registry/data/{name}`\r\n- [X] Check that your yaml is registered at\r\n`evals/registry/evals/{name}.yaml`\r\n- [X] Ensure you have the right to use the data you submit via this eval\r\n\r\n(For now, we will only be approving evals that use one of the existing\r\neval classes. You may still write custom eval classes for your own\r\ncases, and we may consider merging them in the future.)\r\n\r\n## Final checklist 👀\r\n\r\n### Submission agreement\r\n\r\nBy contributing to Evals, you are agreeing to make your evaluation logic\r\nand data under the same MIT license as this repository. You must have\r\nadequate rights to upload any data used in an Eval. OpenAI reserves the\r\nright to use this data in future service improvements to our product.\r\nContributions to OpenAI Evals will be subject to our usual Usage\r\nPolicies (https://platform.openai.com/docs/usage-policies).\r\n\r\n- [X] I agree that my submission will be made available under an MIT\r\nlicense and complies with OpenAI's usage policies.\r\n\r\n### Email address validation\r\n\r\nIf your submission is accepted, we will be granting GPT-4 access to a\r\nlimited number of contributors. Access will be given to the email\r\naddress associated with the merged pull request.\r\n\r\n- [X] I acknowledge that GPT-4 access will only be granted, if\r\napplicable, to the email address used for my merged pull request.\r\n\r\n### Limited availability acknowledgement\r\n\r\nWe know that you might be excited to contribute to OpenAI's mission,\r\nhelp improve our models, and gain access to GPT-4. However, due to the\r\nrequirements mentioned above and high volume of submissions, we will not\r\nbe able to accept all submissions and thus not grant everyone who opens\r\na PR GPT-4 access. We know this is disappointing, but we hope to set the\r\nright expectation before you open this PR.\r\n\r\n- [X] I understand that opening a PR, even if it meets the requirements\r\nabove, does not guarantee the PR will be merged nor GPT-4 access\r\ngranted.\r\n\r\n### Submit eval\r\n\r\n- [X] I have filled out all required fields in the evals PR form\r\n- [ ] (Ignore if not submitting code) I have run `pip install\r\npre-commit; pre-commit install` and have verified that `black`, `isort`,\r\nand `autoflake` are running when I commit and push\r\n\r\nFailure to fill out all required fields will result in the PR being\r\nclosed.\r\n\r\n### Eval JSON data \r\n\r\nSince we are using Git LFS, we are asking eval submitters to add in as\r\nmany Eval Samples (at least 5) from their contribution here:\r\n\r\n<details>\r\n  <summary>View evals in JSON</summary>\r\n\r\n  ### Eval\r\n  ```jsonl\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"university\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"international\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"association\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"individual\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"environmental\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"administration\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"organization\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"associated\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"communication\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"accommodation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"opportunity\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"manufacturer\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"educational\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"evaluation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"implementation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"documentation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"particularly\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"manufacturing\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"administrative\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"immediately\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"configuration\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"multimedia\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"approximately\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"miscellaneous\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"laboratory\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"editorial\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"representative\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"administrator\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"originally\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"participation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"certification\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"contemporary\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"liability\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"agricultural\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"investigation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"elementary\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"organisation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"initiative\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"disability\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"examination\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"consideration\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"biological\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"classification\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"intellectual\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"experimental\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"consolidation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"cooperation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"notification\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"anniversary\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"specification\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"representation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"recommendation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"eventually\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"collaboration\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"operational\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"determination\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"bestiality\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"possibility\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"unfortunately\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"productivity\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"incorporated\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"interpretation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"optimization\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"participating\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"necessarily\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"popularity\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"coordinator\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"electricity\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"undergraduate\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"institutional\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"capability\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"discrimination\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"intermediate\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"preliminary\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"authentication\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"probability\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"orientation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"flexibility\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"occupational\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"modification\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"metropolitan\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"identifying\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"alphabetical\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"bibliography\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"cooperative\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"personality\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"pharmaceutical\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"methodology\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"mathematical\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"constitutional\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"informational\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"coordination\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"affiliated\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"considerable\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"documentary\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"hospitality\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"theoretical\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"evaluated\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"architectural\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"authorization\"}],\"ideal\":\"5\"}\r\n  ```\r\n</details>","shortMessageHtmlLink":"Eval: Syllable count for English words with 5 syllables or more (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1685843688\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/833\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/833/hovercard\" href=\"https://github.com/openai/evals/pull/833\">open…</a>"}},{"before":"fedad26bda506d8a0350d6a65924cb2e32ef46bc","after":"b12ee79f862b2f0b64df35ea1cb44195e1ea0c01","ref":"refs/heads/main","pushedAt":"2023-05-31T05:34:17.607Z","pushType":"push","commitsCount":36,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"Eval: Syllable count for English words with 5 syllables or more (#833)\n\n# Thank you for contributing an eval! ♥️\r\n\r\n🚨 Please make sure your PR follows these guidelines, __failure to follow\r\nthe guidelines below will result in the PR being closed automatically__.\r\nNote that even if the criteria are met, that does not guarantee the PR\r\nwill be merged nor GPT-4 access granted. 🚨\r\n\r\n__PLEASE READ THIS__:\r\n\r\nIn order for a PR to be merged, it must fail on GPT-4. We are aware that\r\nright now, users do not have access, so you will not be able to tell if\r\nthe eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep\r\nin mind as we run the eval, if GPT-4 gets higher than 90% on the eval,\r\nwe will likely reject since GPT-4 is already capable of completing the\r\ntask.\r\n\r\nWe plan to roll out a way for users submitting evals to see the eval\r\nperformance on GPT-4 soon. Stay tuned! Until then, you will not be able\r\nto see the eval performance on GPT-4. **Starting April 10, the minimum\r\neval count is 15 samples, we hope this makes it easier to create and\r\ncontribute evals.**\r\n\r\n## Eval details 📑\r\n### Eval name\r\nsyllables_long_words\r\n\r\n### Eval description\r\n\r\nThe model is given an English word with 5 or more syllables and asked to\r\nstate the number of syllables in the word.\r\n\r\n### What makes this a useful eval?\r\n\r\nKnowing the syllable count of words is critical for producing poetry and\r\nsongs. While ChatGPT (v4) does well with counting syllables for words of\r\nup to 4 syllables, it seems to do poorly with words that have more\r\nsyllables. For my tests on `gpt-3.5-turbo` the model scored less than\r\n70%.\r\n\r\nThis eval is based on part by work done at\r\n[https://github.com/gautesolheim/25000-syllabified-words-list](url)\r\nwhich is freely available under The Unlicense (no restrictions).\r\n\r\n## Criteria for a good eval ✅\r\n\r\nBelow are some of the criteria we look for in a good eval. In general,\r\nwe are seeking cases where the model does not do a good job despite\r\nbeing capable of generating a good response (note that there are some\r\nthings large language models cannot do, so those would not make good\r\nevals).\r\n\r\nYour eval should be:\r\n\r\n- [X] Thematically consistent: The eval should be thematically\r\nconsistent. We'd like to see a number of prompts all demonstrating some\r\nparticular failure mode. For example, we can create an eval on cases\r\nwhere the model fails to reason about the physical world.\r\n- [X] Contains failures where a human can do the task, but either GPT-4\r\nor GPT-3.5-Turbo could not.\r\n- [X] Includes good signal around what is the right behavior. This means\r\neither a correct answer for `Basic` evals or the `Fact` Model-graded\r\neval, or an exhaustive rubric for evaluating answers for the `Criteria`\r\nModel-graded eval.\r\n- [X] **Include at least 15 high quality examples.**\r\n\r\nIf there is anything else that makes your eval worth including, please\r\ndocument it below.\r\n\r\n### Unique eval value\r\n\r\nThis eval contains a very large number of 5 and 6 syllable words in\r\nEnglish, with over 1,500 samples. If desired 7 and 8 syllable words\r\ncould also be added.\r\n\r\n## Eval structure 🏗️\r\n\r\nYour eval should\r\n- [X] Check that your data is in `evals/registry/data/{name}`\r\n- [X] Check that your yaml is registered at\r\n`evals/registry/evals/{name}.yaml`\r\n- [X] Ensure you have the right to use the data you submit via this eval\r\n\r\n(For now, we will only be approving evals that use one of the existing\r\neval classes. You may still write custom eval classes for your own\r\ncases, and we may consider merging them in the future.)\r\n\r\n## Final checklist 👀\r\n\r\n### Submission agreement\r\n\r\nBy contributing to Evals, you are agreeing to make your evaluation logic\r\nand data under the same MIT license as this repository. You must have\r\nadequate rights to upload any data used in an Eval. OpenAI reserves the\r\nright to use this data in future service improvements to our product.\r\nContributions to OpenAI Evals will be subject to our usual Usage\r\nPolicies (https://platform.openai.com/docs/usage-policies).\r\n\r\n- [X] I agree that my submission will be made available under an MIT\r\nlicense and complies with OpenAI's usage policies.\r\n\r\n### Email address validation\r\n\r\nIf your submission is accepted, we will be granting GPT-4 access to a\r\nlimited number of contributors. Access will be given to the email\r\naddress associated with the merged pull request.\r\n\r\n- [X] I acknowledge that GPT-4 access will only be granted, if\r\napplicable, to the email address used for my merged pull request.\r\n\r\n### Limited availability acknowledgement\r\n\r\nWe know that you might be excited to contribute to OpenAI's mission,\r\nhelp improve our models, and gain access to GPT-4. However, due to the\r\nrequirements mentioned above and high volume of submissions, we will not\r\nbe able to accept all submissions and thus not grant everyone who opens\r\na PR GPT-4 access. We know this is disappointing, but we hope to set the\r\nright expectation before you open this PR.\r\n\r\n- [X] I understand that opening a PR, even if it meets the requirements\r\nabove, does not guarantee the PR will be merged nor GPT-4 access\r\ngranted.\r\n\r\n### Submit eval\r\n\r\n- [X] I have filled out all required fields in the evals PR form\r\n- [ ] (Ignore if not submitting code) I have run `pip install\r\npre-commit; pre-commit install` and have verified that `black`, `isort`,\r\nand `autoflake` are running when I commit and push\r\n\r\nFailure to fill out all required fields will result in the PR being\r\nclosed.\r\n\r\n### Eval JSON data \r\n\r\nSince we are using Git LFS, we are asking eval submitters to add in as\r\nmany Eval Samples (at least 5) from their contribution here:\r\n\r\n<details>\r\n  <summary>View evals in JSON</summary>\r\n\r\n  ### Eval\r\n  ```jsonl\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"university\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"international\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"association\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"individual\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"environmental\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"administration\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"organization\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"associated\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"communication\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"accommodation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"opportunity\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"manufacturer\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"educational\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"evaluation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"implementation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"documentation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"particularly\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"manufacturing\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"administrative\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"immediately\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"configuration\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"multimedia\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"approximately\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"miscellaneous\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"laboratory\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"editorial\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"representative\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"administrator\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"originally\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"participation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"certification\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"contemporary\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"liability\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"agricultural\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"investigation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"elementary\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"organisation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"initiative\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"disability\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"examination\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"consideration\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"biological\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"classification\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"intellectual\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"experimental\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"consolidation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"cooperation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"notification\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"anniversary\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"specification\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"representation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"recommendation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"eventually\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"collaboration\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"operational\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"determination\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"bestiality\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"possibility\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"unfortunately\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"productivity\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"incorporated\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"interpretation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"optimization\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"participating\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"necessarily\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"popularity\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"coordinator\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"electricity\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"undergraduate\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"institutional\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"capability\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"discrimination\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"intermediate\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"preliminary\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"authentication\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"probability\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"orientation\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"flexibility\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"occupational\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"modification\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"metropolitan\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"identifying\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"alphabetical\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"bibliography\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"cooperative\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"personality\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"pharmaceutical\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"methodology\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"mathematical\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"constitutional\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"informational\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"coordination\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"affiliated\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"considerable\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"documentary\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"hospitality\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"theoretical\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"evaluated\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"architectural\"}],\"ideal\":\"5\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Please state the number of\r\nsyllables in the input word. Reply only with a number and nothing\r\nelse.\"},{\"role\":\"user\",\"content\":\"authorization\"}],\"ideal\":\"5\"}\r\n  ```\r\n</details>","shortMessageHtmlLink":"Eval: Syllable count for English words with 5 syllables or more (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1685843688\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/833\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/833/hovercard\" href=\"https://github.com/openai/evals/pull/833\">open…</a>"}},{"before":"bc7ccad8afeb0a37d770fc7e19491cad5627bca1","after":"ac8a184bfff956302d09c1b450b3c3bb2f2f4c2d","ref":"refs/heads/GOL","pushedAt":"2023-05-27T02:17:48.823Z","pushType":"push","commitsCount":1,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"revalidate and update samples","shortMessageHtmlLink":"revalidate and update samples"}},{"before":"f335e2f7b47aa93fca595b63c2fd3489a56e57d3","after":"bc7ccad8afeb0a37d770fc7e19491cad5627bca1","ref":"refs/heads/GOL","pushedAt":"2023-05-27T01:00:55.379Z","pushType":"push","commitsCount":1,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"fix lowercasing","shortMessageHtmlLink":"fix lowercasing"}},{"before":"f0dd569003e89d23a7ade02eda3eddf030b37a13","after":"f335e2f7b47aa93fca595b63c2fd3489a56e57d3","ref":"refs/heads/GOL","pushedAt":"2023-05-27T00:49:51.974Z","pushType":"push","commitsCount":104,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"Merge branch 'openai:main' into GOL","shortMessageHtmlLink":"Merge branch 'openai:main' into GOL"}},{"before":"170dfd886c0704588461af075393cc20cfb0480f","after":"fedad26bda506d8a0350d6a65924cb2e32ef46bc","ref":"refs/heads/main","pushedAt":"2023-05-27T00:49:36.909Z","pushType":"push","commitsCount":103,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"[Eval] Portuguese syllable count (#1038)\n\n# Thank you for contributing an eval! ♥️\r\n\r\n🚨 Please make sure your PR follows these guidelines, __failure to follow\r\nthe guidelines below will result in the PR being closed automatically__.\r\nNote that even if the criteria are met, that does not guarantee the PR\r\nwill be merged nor GPT-4 access granted. 🚨\r\n\r\n__PLEASE READ THIS__:\r\n\r\nIn order for a PR to be merged, it must fail on GPT-4. We are aware that\r\nright now, users do not have access, so you will not be able to tell if\r\nthe eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep\r\nin mind as we run the eval, if GPT-4 gets higher than 90% on the eval,\r\nwe will likely reject since GPT-4 is already capable of completing the\r\ntask.\r\n\r\nWe plan to roll out a way for users submitting evals to see the eval\r\nperformance on GPT-4 soon. Stay tuned! Until then, you will not be able\r\nto see the eval performance on GPT-4. **Starting April 10, the minimum\r\neval count is 15 samples, we hope this makes it easier to create and\r\ncontribute evals.**\r\n\r\nAlso, pelase note that we're using **Git LFS** for storing the JSON\r\nfiles, so please make sure that you move the JSON file to Git LFS before\r\nsubmitting a PR. Details on how to use Git LFS are available\r\n[here](https://git-lfs.com).\r\n\r\n## Eval details 📑\r\n### Eval name\r\nPortuguese Syllable Count\r\n\r\n### Eval description\r\n\r\nAsk the model to count how many syllables are in a given word in\r\nPortuguese (more specifically, Brazilian Portuguese), and reply with the\r\nsyllable count.\r\n\r\n### What makes this a useful eval?\r\n\r\nSyllable counting follows pre-determined well-established set of rules\r\nin Portuguese, such as \"ss and rr digraphs are always separated, while\r\nlh, nh and sh are not\", so it should be an easy task for the model, and\r\nit misses the syllable division, resulting in an incorrect syllable\r\ncount.\r\n\r\n## Criteria for a good eval ✅\r\n\r\nBelow are some of the criteria we look for in a good eval. In general,\r\nwe are seeking cases where the model does not do a good job despite\r\nbeing capable of generating a good response (note that there are some\r\nthings large language models cannot do, so those would not make good\r\nevals).\r\n\r\nYour eval should be:\r\n\r\n- [x] Thematically consistent: The eval should be thematically\r\nconsistent. We'd like to see a number of prompts all demonstrating some\r\nparticular failure mode. For example, we can create an eval on cases\r\nwhere the model fails to reason about the physical world.\r\n- [x] Contains failures where a human can do the task, but either GPT-4\r\nor GPT-3.5-Turbo could not.\r\n- [x] Includes good signal around what is the right behavior. This means\r\neither a correct answer for `Basic` evals or the `Fact` Model-graded\r\neval, or an exhaustive rubric for evaluating answers for the `Criteria`\r\nModel-graded eval.\r\n- [x] **Include at least 15 high quality examples.**\r\n\r\n## Eval structure 🏗️\r\n\r\nYour eval should\r\n- [x] Check that your data is in `evals/registry/data/{name}`\r\n- [x] Check that your yaml is registered at\r\n`evals/registry/evals/{name}.yaml`\r\n- [x] Ensure you have the right to use the data you submit via this eval\r\n\r\n(For now, we will only be approving evals that use one of the existing\r\neval classes. You may still write custom eval classes for your own\r\ncases, and we may consider merging them in the future.)\r\n\r\n## Final checklist 👀\r\n\r\n### Submission agreement\r\n\r\nBy contributing to Evals, you are agreeing to make your evaluation logic\r\nand data under the same MIT license as this repository. You must have\r\nadequate rights to upload any data used in an Eval. OpenAI reserves the\r\nright to use this data in future service improvements to our product.\r\nContributions to OpenAI Evals will be subject to our usual Usage\r\nPolicies (https://platform.openai.com/docs/usage-policies).\r\n\r\n- [x] I agree that my submission will be made available under an MIT\r\nlicense and complies with OpenAI's usage policies.\r\n\r\n### Email address validation\r\n\r\nIf your submission is accepted, we will be granting GPT-4 access to a\r\nlimited number of contributors. Access will be given to the email\r\naddress associated with the merged pull request.\r\n\r\n- [x] I acknowledge that GPT-4 access will only be granted, if\r\napplicable, to the email address used for my merged pull request.\r\n\r\n### Limited availability acknowledgement\r\n\r\nWe know that you might be excited to contribute to OpenAI's mission,\r\nhelp improve our models, and gain access to GPT-4. However, due to the\r\nrequirements mentioned above and high volume of submissions, we will not\r\nbe able to accept all submissions and thus not grant everyone who opens\r\na PR GPT-4 access. We know this is disappointing, but we hope to set the\r\nright expectation before you open this PR.\r\n\r\n- [x] I understand that opening a PR, even if it meets the requirements\r\nabove, does not guarantee the PR will be merged nor GPT-4 access\r\ngranted.\r\n\r\n### Submit eval\r\n\r\n- [x] I have filled out all required fields of this form\r\n- [x] I have used **Git LFS** for the Eval JSON data\r\n\r\nFailure to fill out all required fields will result in the PR being\r\nclosed.\r\n\r\n### Eval JSON data \r\n\r\nSince we are using Git LFS, we are asking eval submitters to add in as\r\nmany Eval Samples (at least 5) from their contribution here:\r\n\r\n<details>\r\n  <summary>View evals in JSON</summary>\r\n\r\n  ### Eval\r\n  ```jsonl\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Você lê palavras e retorna o\r\nnúmero de sílabas que ela contém. Retorne apenas o número de sílabas em\r\nalgarismos e nada mais.\"}, {\"role\": \"user\", \"content\": \"pragmático\"}],\r\n\"ideal\": \"4\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Você lê palavras e retorna o\r\nnúmero de sílabas que ela contém. Retorne apenas o número de sílabas em\r\nalgarismos e nada mais.\"}, {\"role\": \"user\", \"content\": \"pictograma\"}],\r\n\"ideal\": \"4\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Você lê palavras e retorna o\r\nnúmero de sílabas que ela contém. Retorne apenas o número de sílabas em\r\nalgarismos e nada mais.\"}, {\"role\": \"user\", \"content\": \"discente\"}],\r\n\"ideal\": \"3\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Você lê palavras e retorna o\r\nnúmero de sílabas que ela contém. Retorne apenas o número de sílabas em\r\nalgarismos e nada mais.\"}, {\"role\": \"user\", \"content\": \"escolástico\"}],\r\n\"ideal\": \"5\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Você lê palavras e retorna o\r\nnúmero de sílabas que ela contém. Retorne apenas o número de sílabas em\r\nalgarismos e nada mais.\"}, {\"role\": \"user\", \"content\": \"cegonha\"}],\r\n\"ideal\": \"3\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Você lê palavras e retorna o\r\nnúmero de sílabas que ela contém. Retorne apenas o número de sílabas em\r\nalgarismos e nada mais.\"}, {\"role\": \"user\", \"content\": \"linha\"}],\r\n\"ideal\": \"2\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Você lê palavras e retorna o\r\nnúmero de sílabas que ela contém. Retorne apenas o número de sílabas em\r\nalgarismos e nada mais.\"}, {\"role\": \"user\", \"content\": \"beterraba\"}],\r\n\"ideal\": \"4\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Você lê palavras e retorna o\r\nnúmero de sílabas que ela contém. Retorne apenas o número de sílabas em\r\nalgarismos e nada mais.\"}, {\"role\": \"user\", \"content\": \"ódio\"}],\r\n\"ideal\": \"2\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Você lê palavras e retorna o\r\nnúmero de sílabas que ela contém. Retorne apenas o número de sílabas em\r\nalgarismos e nada mais.\"}, {\"role\": \"user\", \"content\": \"duelo\"}],\r\n\"ideal\": \"3\"}\r\n{\"input\": [{\"role\": \"system\", \"content\": \"Você lê palavras e retorna o\r\nnúmero de sílabas que ela contém. Retorne apenas o número de sílabas em\r\nalgarismos e nada mais.\"}, {\"role\": \"user\", \"content\": \"apocalíptico\"}],\r\n\"ideal\": \"6\"}\r\n  ```\r\n</details>","shortMessageHtmlLink":"[Eval] Portuguese syllable count (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1726363475\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/1038\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/1038/hovercard\" href=\"https://github.com/openai/evals/pull/1038\">openai#1038</a>)"}},{"before":"07a02c70c024970c695f340d3dace34de896c4e7","after":"f0dd569003e89d23a7ade02eda3eddf030b37a13","ref":"refs/heads/GOL","pushedAt":"2023-05-27T00:46:15.609Z","pushType":"push","commitsCount":149,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"remove samples-mini, lowercased GOL","shortMessageHtmlLink":"remove samples-mini, lowercased GOL"}},{"before":"0ad0f4d2886ca492eb49f642eb98b90cbbd8de50","after":"170dfd886c0704588461af075393cc20cfb0480f","ref":"refs/heads/main","pushedAt":"2023-05-09T19:28:51.466Z","pushType":"push","commitsCount":146,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"[Eval] An array of Liar Paradox-based evals (#883)\n\n# Thank you for contributing an eval! ♥️\r\n\r\n🚨 Please make sure your PR follows these guidelines, __failure to follow\r\nthe guidelines below will result in the PR being closed automatically__.\r\nNote that even if the criteria are met, that does not guarantee the PR\r\nwill be merged nor GPT-4 access granted. 🚨\r\n\r\n__PLEASE READ THIS__:\r\n\r\nIn order for a PR to be merged, it must fail on GPT-4. We are aware that\r\nright now, users do not have access, so you will not be able to tell if\r\nthe eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep\r\nin mind as we run the eval, if GPT-4 gets higher than 90% on the eval,\r\nwe will likely reject since GPT-4 is already capable of completing the\r\ntask.\r\n\r\nWe plan to roll out a way for users submitting evals to see the eval\r\nperformance on GPT-4 soon. Stay tuned! Until then, you will not be able\r\nto see the eval performance on GPT-4. **Starting April 10, the minimum\r\neval count is 15 samples, we hope this makes it easier to create and\r\ncontribute evals.**\r\n\r\n## Eval details 📑\r\n### Eval name\r\nlogic-liar-paradox\r\n\r\n### Eval description\r\n\r\nAn array of Liar Paradox-based evals, examining the model's proficiency\r\nin navigating linguistic nuances and logical reasoning within\r\nself-referential statements.\r\n\r\n### What makes this a useful eval?\r\n\r\nThis eval is particularly useful because it delves into complex, nuanced\r\nlogical concepts and self-referential statements, which have\r\nhistorically posed challenges for AI models. By exploring various\r\ncontexts, alternative logical frameworks, and modifications to\r\nstatements, this eval helps assess the model's ability to adapt to\r\ndifferent perspectives, grasp subtleties in language, and engage in\r\nflexible reasoning. The ability to understand and navigate paradoxes is\r\nan essential aspect of human-like reasoning, and improving an AI model's\r\nperformance in this area would significantly enhance its overall\r\nusefulness and reliability in real-world applications. Additionally,\r\nshowcasing the model's improved proficiency in handling paradoxes would\r\nnot only make for a compelling marketing angle (as paradoxes are\r\nunderstood by a much broader range of people than other difficult tasks\r\nsuch as pure maths or quantum mechanics) but it would also demonstrate\r\nthe progress made in AI's capacity to think and reason more like humans.\r\nIt also adds paradox-absorbing crumple zones.\r\n\r\n## Criteria for a good eval ✅\r\n\r\nBelow are some of the criteria we look for in a good eval. In general,\r\nwe are seeking cases where the model does not do a good job despite\r\nbeing capable of generating a good response (note that there are some\r\nthings large language models cannot do, so those would not make good\r\nevals).\r\n\r\nYour eval should be:\r\n\r\n- [x] Thematically consistent: The eval should be thematically\r\nconsistent. We'd like to see a number of prompts all demonstrating some\r\nparticular failure mode. For example, we can create an eval on cases\r\nwhere the model fails to reason about the physical world.\r\n- [x] Contains failures where a human can do the task, but either GPT-4\r\nor GPT-3.5-Turbo could not.\r\n- [x] Includes good signal around what is the right behavior. This means\r\neither a correct answer for `Basic` evals or the `Fact` Model-graded\r\neval, or an exhaustive rubric for evaluating answers for the `Criteria`\r\nModel-graded eval.\r\n- [x] **Include at least 15 high quality examples.**\r\n\r\nIf there is anything else that makes your eval worth including, please\r\ndocument it below.\r\n\r\n- [x] Addresses complex logical reasoning: The eval focuses on AI's\r\nability to comprehend and navigate paradoxes, self-referential\r\nstatements, and context switching, which are important aspects of\r\nhuman-like reasoning. By testing the model's proficiency in these areas,\r\nwe can identify areas for improvement and work towards enhancing AI's\r\noverall capacity to think and reason more like humans.\r\n- [x] Demonstrates adaptability and flexibility: The eval showcases the\r\nmodel's ability to switch between contexts, alter premises, and engage\r\nwith different dimensions of inferred logic. This will help assess the\r\nmodel's adaptability and flexibility in diverse real-world situations,\r\nmaking it more reliable and useful.\r\n- [x] Contributes to AI safety and understanding: By identifying the\r\nmodel's weaknesses and limitations in handling paradoxes and complex\r\nlogical constructs, the eval can contribute to AI safety and enable\r\nresearchers to better understand the challenges faced by large language\r\nmodels in these areas.\r\n- [x] Engaging and appealing: An eval that delves into paradoxes and\r\ncomplex thought exercises is not only intellectually stimulating but\r\nalso adds an appealing element to showcase the model's capabilities,\r\nmaking it more attractive for both researchers and end-users.\r\n\r\n### Unique eval value\r\n\r\n- [x] Encourages creativity and lateral thinking: The eval, by focusing\r\non paradoxes and complex logical constructs, encourages both the AI and\r\nits developers to think creatively and approach problem-solving from\r\nunconventional angles. This can lead to the discovery of novel solutions\r\nand a better understanding of the model's capabilities.\r\n- [x] Aligns with human values and expectations: An AI that can\r\nsuccessfully navigate paradoxes and complex logic is more likely to\r\nalign with human values and expectations. By addressing these challenges\r\nin the eval, we strive to develop AI systems that understand and respect\r\nthe nuances of human thought and communication.\r\n- [x] Addresses a broad range of applications: Improved reasoning and\r\ncontext-switching abilities can have a significant impact on various AI\r\napplications, including natural language understanding, decision-making,\r\nand problem-solving in domains such as law, philosophy, ethics, and\r\nmore.\r\n- [x] Fosters interdisciplinary collaboration: The exploration of\r\nparadoxes and complex logic often draws upon insights from multiple\r\ndisciplines, including philosophy, linguistics, psychology, and computer\r\nscience. This eval can help foster interdisciplinary collaboration,\r\nleading to richer and more diverse perspectives on AI development and\r\nits potential impact on society.\r\n\r\n## Eval structure 🏗️\r\n\r\nYour eval should\r\n- [x] Check that your data is in `evals/registry/data/{name}`\r\n- [x] Check that your yaml is registered at\r\n`evals/registry/evals/{name}.yaml`\r\n- [x] Ensure you have the right to use the data you submit via this eval\r\n\r\n(For now, we will only be approving evals that use one of the existing\r\neval classes. You may still write custom eval classes for your own\r\ncases, and we may consider merging them in the future.)\r\n\r\n## Final checklist 👀\r\n\r\n### Submission agreement\r\n\r\nBy contributing to Evals, you are agreeing to make your evaluation logic\r\nand data under the same MIT license as this repository. You must have\r\nadequate rights to upload any data used in an Eval. OpenAI reserves the\r\nright to use this data in future service improvements to our product.\r\nContributions to OpenAI Evals will be subject to our usual Usage\r\nPolicies (https://platform.openai.com/docs/usage-policies).\r\n\r\n- [x] I agree that my submission will be made available under an MIT\r\nlicense and complies with OpenAI's usage policies.\r\n\r\n### Email address validation\r\n\r\nIf your submission is accepted, we will be granting GPT-4 access to a\r\nlimited number of contributors. Access will be given to the email\r\naddress associated with the merged pull request.\r\n\r\n- [x] I acknowledge that GPT-4 access will only be granted, if\r\napplicable, to the email address used for my merged pull request.\r\n\r\n### Limited availability acknowledgement\r\n\r\nWe know that you might be excited to contribute to OpenAI's mission,\r\nhelp improve our models, and gain access to GPT-4. However, due to the\r\nrequirements mentioned above and high volume of submissions, we will not\r\nbe able to accept all submissions and thus not grant everyone who opens\r\na PR GPT-4 access. We know this is disappointing, but we hope to set the\r\nright expectation before you open this PR.\r\n\r\n- [x] I understand that opening a PR, even if it meets the requirements\r\nabove, does not guarantee the PR will be merged nor GPT-4 access\r\ngranted.\r\n\r\n### Submit eval\r\n\r\n- [x] I have filled out all required fields in the evals PR form\r\n- [ ] (Ignore if not submitting code) I have run `pip install\r\npre-commit; pre-commit install` and have verified that `black`, `isort`,\r\nand `autoflake` are running when I commit and push\r\n\r\nFailure to fill out all required fields will result in the PR being\r\nclosed.\r\n\r\n### Eval JSON data \r\n\r\nSince we are using Git LFS, we are asking eval submitters to add in as\r\nmany Eval Samples (at least 5) from their contribution here:\r\n\r\n<details>\r\n  <summary>View evals in JSON</summary>\r\n\r\n  ### Eval\r\n  ```jsonl\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Is the statement \\\"This sentence\r\nis true\\\" a paradox?\"}],\"ideal\":\"No, \\\"This sentence is true\\\" is not a\r\nparadox; it is a tautology.\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Does the statement \\\"This sentence\r\nis not true\\\" create a paradox?\"}],\"ideal\":\"Yes, \\\"This sentence is not\r\ntrue\\\" creates a paradox, as it is a variant of the Liar Paradox.\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"What is the difference between a\r\nparadox and a tautology?\"}],\"ideal\":\"A paradox is a statement that leads\r\nto a self-contradictory or logically unacceptable conclusion, while a\r\ntautology is a statement that is always true by its logical form.\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Can the Liar Paradox be resolved\r\nby assuming that sentences can have both true and false\r\nvalues?\"}],\"ideal\":\"No, the Liar Paradox cannot be resolved by assuming\r\nthat sentences can have both true and false values, as this would lead\r\nto a different kind of paradox called the \\\"Dialetheism Paradox.\\\"\"}\r\n{\"input\":[{\"role\":\"system\",\"content\":\"Consider the statement \\\"This\r\nsentence is neither true nor false.\\\" Is this statement an example of\r\nthe Liar Paradox?\"}],\"ideal\":\"This statement, \\\"This sentence is neither\r\ntrue nor false,\\\" is not an example of the Liar Paradox, but it is a\r\nsimilar paradox known as the 'truth-teller paradox' or the 'strengthened\r\nliar paradox.' It creates a paradoxical situation because if the\r\nstatement is true, then it is neither true nor false, which contradicts\r\nits truth. If the statement is false, then it is not the case that it is\r\nneither true nor false, which implies that it is either true or false,\r\nagain leading to a contradiction. The paradox arises due to\r\nself-reference and the inability to assign a consistent truth value to\r\nthe statement.\"}\r\n  ```\r\n</details>","shortMessageHtmlLink":"[Eval] An array of Liar Paradox-based evals (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1690604730\" data-permission-text=\"Title is private\" data-url=\"https://github.com/openai/evals/issues/883\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/openai/evals/pull/883/hovercard\" href=\"https://github.com/openai/evals/pull/883\">openai#883</a>)"}},{"before":"2f2a911f76bfd4c1ed02220f7d2f8834d3761309","after":"0ad0f4d2886ca492eb49f642eb98b90cbbd8de50","ref":"refs/heads/main","pushedAt":"2023-03-19T07:45:00.107Z","pushType":"push","commitsCount":1,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"Update PULL_REQUEST_TEMPLATE.md","shortMessageHtmlLink":"Update PULL_REQUEST_TEMPLATE.md"}},{"before":"50a375beea47718dd35309699a9b0a4633bfe800","after":"07a02c70c024970c695f340d3dace34de896c4e7","ref":"refs/heads/GOL","pushedAt":"2023-03-19T07:43:47.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"arrray output and bump version","shortMessageHtmlLink":"arrray output and bump version"}},{"before":null,"after":"50a375beea47718dd35309699a9b0a4633bfe800","ref":"refs/heads/GOL","pushedAt":"2023-03-19T06:34:28.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"add more robust test data","shortMessageHtmlLink":"add more robust test data"}},{"before":"2874930e00cf9b9525e9ebc8ba91fc866d28bbc1","after":"99c889dca1b0188a15fbcaa7690924d472fee9b3","ref":"refs/heads/syntax-check","pushedAt":"2023-03-19T00:43:30.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"data update","shortMessageHtmlLink":"data update"}},{"before":null,"after":"2874930e00cf9b9525e9ebc8ba91fc866d28bbc1","ref":"refs/heads/syntax-check","pushedAt":"2023-03-19T00:04:30.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"AaronGoldsmith","name":"Aaron Goldsmith","path":"/AaronGoldsmith","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/16547926?s=80&v=4"},"commit":{"message":"initial commit with data","shortMessageHtmlLink":"initial commit with data"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAADSmhkqwA","startCursor":null,"endCursor":null}},"title":"Activity · AaronGoldsmith/evals"}