Randomly select MMMU answer when none is returned from the model (ope…

…nai#1447)

This is the behavior MMMU used for evaluating, so we should match this
here.

As an example this increased the mmmu-music benchmark from `0.3666` to
`0.4` as multiple questions in that benchmark were unanswered by the
model

evals/elsuite/mmmu/eval.py

@@ -159,6 +159,14 @@ def eval_sample(self, sample: Sample, rng):
 
  match = sampled.find(f"ANSWER: {correct_answer}") != -1
 
+ if not match and sampled.find("ANSWER") == -1 and sample.question_type == "multiple-choice":
+ # The model didn't answer anything, so randomly pick an answer
+ # This matches the behavior described in section 4.1 of the MMMU paper: https://arxiv.org/pdf/2311.16502.pdf
+ logging.info("No answer found for multiple choice so picking a random answer.")
+ answer_idx = rng.randint(0, len(sample.answers) - 1)
+ answer_letter = chr(ord("A") + answer_idx)
+ match = correct_answer == answer_letter
+
  record_match(
  match,
  expected=correct_answer,