Support for top-k metrics with num_return_sequences>1

[yuyemin]

Hi, I wonder if lm-eval supports top-k evaluation when generating with beam_search with num_return_sequences>1 ?

[haileyschoelkopf]

Hi! We don't currently support num_return_sequences > 1, and mostly focus on support for Top-P / Top-K temperature sampling, though we do support running generation K times per document to get K different completions via repeats: K in a task YAML.

Could you explain exactly what "top-K evaluation" means in this context?

If you'd be interested in opening a PR to help support this or streamline evaluation in the num_return_sequences > 1 / beam search setting we'd be happy to accept it!

[yuyemin]

Thanks for the quick reply! I'm aiming to evaluate the top-k exact-match metric (check if there's a ground-truth hit in top-k lowest-perplexity candidates generated using beam-search + num_return_sequences=k).

In that case, I'll see if I can work on incorporating this feature and open a PR afterwards.

Hi! We don't currently support num_return_sequences > 1, and mostly focus on support for Top-P / Top-K temperature sampling, though we do support running generation K times per document to get K different completions via repeats: K in a task YAML.

Could you explain exactly what "top-K evaluation" means in this context?

If you'd be interested in opening a PR to help support this or streamline evaluation in the num_return_sequences > 1 / beam search setting we'd be happy to accept it!

[ChengJade]

Thanks for the quick reply! I'm aiming to evaluate the top-k exact-match metric (check if there's a ground-truth hit in top-k lowest-perplexity candidates generated using beam-search + num_return_sequences=k).

In that case, I'll see if I can work on incorporating this feature and open a PR afterwards.

Hi! We don't currently support num_return_sequences > 1, and mostly focus on support for Top-P / Top-K temperature sampling, though we do support running generation K times per document to get K different completions via repeats: K in a task YAML.
Could you explain exactly what "top-K evaluation" means in this context?
If you'd be interested in opening a PR to help support this or streamline evaluation in the num_return_sequences > 1 / beam search setting we'd be happy to accept it!

Hi, sorry to disturb you, just wondering have you worked out this?