[WIP] Add chat templating for HF models

[haileyschoelkopf]

This is a WIP PR , carrying on the draft @daniel-furman in #1209 started of adding the specified oft-requested chat templating feature.

Current TODOs are:

• Check performance using e.g. OpenHermes and Llama-2-7b-chat-hf (+ your chat model here)
• expose CLI flags for sysprompt + toggling on chat template
• (?) turn this into a mixin and enable it for vLLM, etc.?
• Decide what user-controllable functionality needs to be added on top of this (e.g. how to deal with few-shot examples in context, how to add "triggers" that are added after the assistant's last response, ...)
• ...

Feedback appreciated!

[CLAassistant]

All committers have signed the CLA.

[daniel-furman]

Awesome to see this PR go up! Happy to provide context (pun intended) on some of my commits, my experimentation was starting to get messy at the end there.

Overall, I wasn't seeing the positive move in eval scores that I had anticipated. I hypothesize that dealing with few-shot examples in context and adding triggers will be helpful to getting scores to be better.

[haileyschoelkopf]

Likewise, initial tests with OpenHermes on my end are showing similarly, on ARC and Lambada (the latter of which does not surprise me too much).

We will also need to address the correctness of target_delimiter remaining the same when handling chat templates.

[haileyschoelkopf]

Yep, I have a more recent version I’ll push tomorrow morning!

[tmabraham]

does this currently not allow for custom system prompts?

[haileyschoelkopf]

TODOs:

• get numbers on generative tasks with + without chat templating
• Work out how to handle delimiters / spacing for multiple_choice tasks
• Confirm this interacts alright with models' max length
• .........
• Ensure this raises an error when a model doesn't set a chat template in its tokenizer config
• also port to vllm LM class
• Decide if this is the best place to edit the codebase with these changes

[sanchit-ahuja]

Hi @haileyschoelkopf, have the changes been made part of the current main branch? If not, what are the blockers for this? Should I start tackling the TODOs that you have mentioned above for this PR?
Happy to help!

[haileyschoelkopf]

courtesy of @KonradSzafer @clefourrier , we've just merged #1873 , supporting chat templating in HF models -- feedback on additional features or blind spots there would be very greatly appreciated!