llama3-base gsm8k score

[rangehow]

I met a problem relevant to #1799.
The scores of LLaMA3-8B-base on the GSM8K benchmark are significantly lower than the scores reported by the official sources.

I use gsm8k_cot task in lm_eval_harness get same score 50+ as #1799 (comment) while official reported is 79.6

Any idea about this ?

[haileyschoelkopf]

The 79.6 GSM8k number reported by Meta comes from their instruct model. I'm not certain if they've reported "official" GSM8k base model scores

[rangehow]

Sorry for not paying attention to the subtitle， yes 79.6 is llama3-8B-it score : )