Hello, I would like to know if there is a method to use "generate_until" to evaluate on the ceval or cmmlu dataset. I'm using a chat model, which adds a prompt template to make it answer questions. However, the model's answer choices (like A, B, C, D) may not necessarily be the first generated token.

[noforit]

No description provided.

[haileyschoelkopf]

Hi!

We don't currently have a generative counterpart for all loglikelihood / multiple choice tasks in a one-size-fits-all way.

However, for CEval or CMMLU, you could make a variant that is zero-shot CoT, following _mmlu_flan_cot_zeroshot_template_yaml in this PR: https://github.com/EleutherAI/lm-evaluation-harness/pull/1356/files#diff-77e4a56ffcde165e312239a2abcef4943035c61df2eb09249d8872ce91f29658