Skip to content

Commit

Permalink
minor: fix one answer in sample logic eval (openai#504)
Browse files Browse the repository at this point in the history
Just fixes the logic puzzle answer, which was a bit wrong. Close
openai#164.
  • Loading branch information
DustinWehr committed Mar 30, 2023
1 parent 10f1dc2 commit f097a2f
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
2 changes: 1 addition & 1 deletion evals/registry/data/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Another example of a Match eval is:
{"input":[{"role":"system","content":"Two people want to cross a river. The only way to get across is with a boat that they find on one side; but that boat can only take one person at a time. The boat cannot return on its own, and there are no ropes to haul it back, yet both persons manage to cross using the boat. How did they do it?"}],"ideal":"The people are on different sides of the river, so the person on the same side as the boat originally can cross first to bring the boat to the side with the other person, then that person can cross."}
{"input":[{"role":"system","content":"There are two men. One of them is wearing a red shirt, and the other is wearing a blue shirt. The two men are named Andrew and Bob, but we do not know which is Andrew and which is Bob. The guy in the blue shirt says, 'I am Andrew.' The guy in the red shirt says, 'I am Bob.' If we know that at least one of them lied, then what color shirt is Andrew wearing?"}],"ideal":"Andrew is wearing the red shirt."}
{"input":[{"role":"system","content":"Which word does NOT belong with the others? A. index B. glossary C. chapter D. book"}],"ideal":"D. book"}
{"input":[{"role":"system","content":"The day before yesterday, Chris was 7 years old. Next year he'll turn 10. How is this possible?"}],"ideal":"Assuming today is January 1st of any given year: Two days ago, on December 30th, Chris was 7 years old. On December 31st, Chris celebrated his 8th birthday. On December 31st of this year, Chris will celebrate his 9th birthday. By the end of this year, on December 31st, Chris will be 10 years old. Therefore, on January 1st of the following year, Chris will be 10 years old."}
{"input":[{"role":"system","content":"The day before yesterday, Chris was 7 years old. Next year he'll turn 10. How is this possible?"}],"ideal":"Assuming today is January 1st of any given year: Two days ago, on December 30th, Chris was 7 years old. The next day, on December 31st, Chris celebrated his 8th birthday. On December 31st of this year, Chris will celebrate his 9th birthday. At the end of next year, on December 31st, Chris will turn 10 years old."}
{"input":[{"role":"system","content":"Inhabitants of an island lie consistently on Tuesdays, Thursdays, and Saturdays, and they tell the truth on the other four days of the week. You have forgotten what day of the week it is, so you ask a passerby. 'Saturday,' he answers. 'And what day will it be tomorrow?' you inquire. 'Wednesday,' he replies. Can you tell what day it is today?"}],"ideal":"Based on answer 1, today cannot be M, W, F, Su, or Sa (lying day). Based on answer 2, today cannot be M, W, F, Su, or Tu (lying day). So, today must be Thursday."}
{"input":[{"role":"system","content":"You are on an island populated by two tribes. Members of one tribe consistently lie. Members of the other tribe always tell the truth. Tribe members can recognize one another, but you can't tell them apart. You meet two people, C and D on the island. C says, 'Exactly one of us is from the liars tribe.' Which tribe is D from?"}],"ideal":"D is from the Liars tribe."}
{"input":[{"role":"system","content":"There are five people in a room. Each person will either always tell the truth or always tell a lie. Each person is asked the following question: How many liars are among you? The answers are: \"one\", \"two\", \"three\", \"four\", \"five\". How many liars are in the room?"}],"ideal":"There are four liars."}
Expand Down
4 changes: 2 additions & 2 deletions evals/registry/data/logic/samples.jsonl
Git LFS file not shown

0 comments on commit f097a2f

Please sign in to comment.