Skip to content

Commit

Permalink
Merge pull request anthropics#12 from eltociear/patch-1
Browse files Browse the repository at this point in the history
Update building_evals.ipynb
  • Loading branch information
alexalbertt committed Mar 11, 2024
2 parents 126b05a + 7367e82 commit 36bef6b
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion misc/building_evals.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -374,7 +374,7 @@
"Now you know about different grading design patterns for evals, and are ready to start building your own. As you do, here are a few guiding pieces of wisdom to get you started.\n",
"- Make your evals specific to your task whenever possible, and try to have the distribution in your eval represent ~ the real life distribution of questions and question difficulties.\n",
"- The only way to know if a model-based grader can do a good job grading your task is to try. Try it out and read some samples to see if your task is a good candidate.\n",
"- Often all that lies between you and an automatable eval is clever design. Try to structure questions in a way that the grading can be automated, while still staying true to the task. Reformatting questions into multipe choice is a common tactic here.\n",
"- Often all that lies between you and an automatable eval is clever design. Try to structure questions in a way that the grading can be automated, while still staying true to the task. Reformatting questions into multiple choice is a common tactic here.\n",
"- In general, your preference should be for higher volume and lower quality of questions over very low volume with high quality."
]
}
Expand Down

0 comments on commit 36bef6b

Please sign in to comment.