Add chat template

[KonradSzafer]

This pull request adds a chat template that allows a few examples to be provided as a conversation between user and assistant, or as a single message from the user, as well as providing a system prompt.

@clefourrier @NathanHB

[clefourrier]

@KonradSzafer fix the linting please

[haileyschoelkopf]

Thanks very much @KonradSzafer for bearing with me!

had a couple other small comments about edge cases, but in general, looks really great!

[haileyschoelkopf]

Final round of comments! Will also add a docs update to docs/model_guide.md imminently.

[LSinev]

May I suggest template actually used to be reported in some artifacts from run (with or without --log-samples used) and also the system instruction.

[KonradSzafer]

May I suggest template actually used to be reported in some artifacts from run (with or without --log-samples used) and also the system instruction.

Hi @LSinev! Good point, updated the branch with the changes!

[haileyschoelkopf]

LGTM! Thanks for all your work on this :)

[djstrong]

I have tested our model with the template. Without template:


With chat template:

Avg g - average of generate_until versions of tasks
Avg mc - average of multiple_choice versions of tasks

I will evaluate more models.

[djstrong]

What do you think, models should be tested with argument fewshot_as_multiturn or without?

[KonradSzafer]

Hi @djstrong!
fewshot_as_multiturn tends to result in lower scores, but is more reflective of the real-world use of the model.
For learning more about the differences in results when using the chat template, I recommend reading this disscusion.

[djstrong]

I tested this only for one model and I confirm that with fewshot_as_multiturn scores are mostly lower.

I have tested dozen of models with and without chat templates but need to analyze results. Available here: https://huggingface.co/spaces/speakleash/open_pl_llm_leaderboard with ,chat sufix.