-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactored v0.4 version shows differences to the existing harness in Japanese. #1392
Comments
Using one sample
options: [ to show input differences when computing loglikelihood. For the first option |
If the Stability AI fork is widely used by the Japanese LLM community, we should prioritize the formatting used there. In the case of However, in the case of the xwinograd implementation, I'm less certain about how to proceed. The Stability AI fork implements it in a fashion that's contradictory to widely accepted practice in the English LLM community. The way we do it is how pretty much every LLM paper going back to (and before) GPT-3 does it for autoregressive language models. Specifically, we compute @leocnj are there other tasks where there are differences? Have you had an opportunity to verify the others are the same? |
Now
lm-eval v0.4
contains several multilingua tasks, e.g., mgsm, xwinograd, and so on.For Japanese (Ja), a widely used LLM eval tool is a forked
lm-eval
(version < 0.4) by Stability-AI https://github.com/Stability-AI/lm-evaluation-harness/tree/jp-stable. Inside, it defines a suite of Ja tasks. This tool will be called asharness_Ja
in this issue report from now on.I just compared
lm-eval v0.4
withharness_Ja
on the two tasks, i.e., mgsm and xwinograd.lm-eval v0.4
result:hf (pretrained=cyberagent/open-calm-1b,dtype=float16), gen_kwargs: (None), limit: 200.0, num_fewshot: None, batch_size: auto (64)
harness_Ja
result:hf-causal (pretrained=cyberagent/open-calm-1b), limit: [200, 200], num_fewshot: [0, 0], batch_size: 32
Some differences can be observed:
mgsm
task,exact_match
is now used, different toacc
in theharness_Ja
.xwinograd
task, results are different, 0.59 (lm-eval v0.4
) vs. 0,.66 (harness_Ja
).Regarding the difference 2, it looks that loglikelihood computing are arranged differently in these two versions. For
harness_Ja
, the computing is on entire sentences. However, inlm-eval v0.4
, we only use a partial sentence.The text was updated successfully, but these errors were encountered: