Some confusion about eval

[willer-lu]

For example, if the ground truth is 'University Yale', but the output of llm is 'Yale University', or '01.01' vs '1st January'.
Such instance should be considered right or wrong when caculating EM?

[GasolSun36]

Hi,
It's definitely wrong, so this is a shortcoming of EM. However, you can use turbo to determine whether the calculation is correct or incorrect given the ground-truth and the generated result.