Different batch sizes give different results

[sedrick-keh-tri]

accelerate launch -m lm_eval --model hf --model_args pretrained=mistralai/Mistral-7B-v0.1 --tasks hellaswag --batch_size 1

gives

hf (pretrained=mistralai/Mistral-7B-v0.1), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 1
|  Tasks  |Version|Filter|n-shot| Metric |Value |   |Stderr|
|---------|------:|------|-----:|--------|-----:|---|-----:|
|hellaswag|      1|none  |     0|acc     |0.6128|±  |0.0049|
|         |       |none  |     0|acc_norm|0.8110|±  |0.0039|

while

accelerate launch -m lm_eval --model hf --model_args pretrained=mistralai/Mistral-7B-v0.1 --tasks hellaswag --batch_size 16

gives

hf (pretrained=mistralai/Mistral-7B-v0.1), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 16
|  Tasks  |Version|Filter|n-shot| Metric |Value |   |Stderr|
|---------|------:|------|-----:|--------|-----:|---|-----:|
|hellaswag|      1|none  |     0|acc     |0.6124|±  |0.0049|
|         |       |none  |     0|acc_norm|0.8106|±  |0.0039|

Here the difference is quite small, but I remember trying some other cases there before where the difference is somewhat big. Why do different batch sizes give different results? Is this the intended behavior?

[haileyschoelkopf]

#1625 has some related discussion!

TL;DR: full determinism on GPUs can be hard! If you see differences in score greater than this or greater than the stderrs, definitely file another issue.