Only a single filtered_resps is logged for repeat > 1 for each sample

[baberabb]

For repeat > 1, model outputs (resps) for each model call are saved to file when using --log_samples but only a single filtered_resp (the first one?).

[haileyschoelkopf]

Is this independent of the take-first filter being the default setting?

[baberabb]

This was without setting an explicit filter in the yaml while testing predict_only. Also doesn't just picking the first generation defeat the whole point of repeat? Why is it even a thing. I'm probably missing something obvious.

[haileyschoelkopf]

repeat is a thing for the purpose of things like Maj@K / self-consistency--if someone wants to use this, they'll set repeats: K and explicitly set some other filter, such that filtered_resps will end up with one response, namely the most-frequently outputted answer.

The idea is that resps contains all K outputs from the model, and the filter pipelines distill it into a single output that can be scored. (or in multiple_choice's case, a single set of N_CHOICES outputs.) This could plausibly be improved or made more easy to understand in the output files for sure and maybe we do want a different default behavior for repeats > 1.

[baberabb]

Aah that makes more sense. Wouldn't pass@k be more appropriate for the default? IMO that's a bit more intuitive ( + warning if just greedy sampling).