Does DSPy have a way to ensure the best chat templates are being used?

[Sciumo]

According to a recent tweet applying appropriate template can have a significant impact.
Does DSPy have a way to ensure the best templates are being used?

[RLinnae]

I am assuming it does not as I had extremely poor results with nous Hermes 7b dpo.

I was actually hoping that DSPy could help with the template optimization as recent papers have shown that even small variance like a space after <> etc can have significant impact on the results.

[thomasnormal]

This is super interesting!
Related to #662
@KCaverly, @okhat, @CyrusOfEden what do you think?

I think it would be great if Predict had a template instance, similar to how it has a signature.
This could be set by the user, based on their personal experience with the LM, or optimized similarly to how signatures get optimized.

Alternatively each LM subclass could define it's own "best template" optimized by us.

[thomasahle]

Maybe somebody could create an awesome "JinjaTemplate" that automatically figures out how to extract.
Or creating the extractor function could simply be part of the task for the LLM doing the template optimization.
Or it could just be a bayesian / optuna optimization over 2-5 predefined templates.

We don't really have to worry about all that though.
As long as the backend supports custom templates, which can be classes with generate and extract methods, then people can make cool implementations later.

[isaacbmiller]

This is something that @okhat and I had a conversation about this week. I was going to do this work sometime this weekend.

I was going to work on creating:

1. A default chat/instruct template.
2. A set of basic benchmarks to figure out which of the templates to use per LM
3. A "jinjaTemplate" or something of similar nature that will just try a few different variants

Open for someone else to take the lead here if they want it to be implemented sooner than Monday ish

[isaacbmiller]

@KCaverly I was potentially going to work off of the backend-refactor branch? Do you have thoughts on that

[Sciumo]

Exactly. Which is what I'm working on.
Also why I don't use DSPy currently. I've been working with methods of sufficiency and acceptance since 2004.
Very difficult to automate, and so Human-in-the-Loop is required, IMO.
Therefore the metric methodology of DSPy doesn't fit my architecture.

[KCaverly]

@isaacbmiller i think you would be fine to work off the backend-refactor branch, my latest PR is not merged yet and includes a lot of changes, but shouldnt affect the Template too much. If you have any questions feel free to reach out.

[Wintoplay]

Hope that LLama.cpp + instruction template [INST][/INST] and chat template will be supported

[Sciumo]

Opus also very much likes XML tags.
Sprinkling them in as emphasis appears to help.
DSPy will not do this.

[KCaverly]

You will have direct access to your prompt templates with the new backend refactor. We will likely have a few defaults and users will be free to tweak their prompts as they feel like. Over time, I hope we can find a better abstraction for this and offer more customization in a simpler way.

[Wintoplay]

That sounds very nice.👍

Notify us when it come to main branch.

[Sciumo]

getting functools to work reliably stress tests template sensitivity analysis.

[tom-doerr]

Is there an update on this? I'm getting poor results with Llama-3-70B-Instruct

[Sciumo]

Interestingly, "LLMs demonstrate a noticeable sensitivity to the evidence order".
Now, this is something that DSPy actually can help, but that requires an Order of Evidence metric system for reordering CoT.
https://arxiv.org/pdf/2305.13300

[maziyarpanahi]

I was wondering about this as well. I mostly use local LLMs, so it has always been a question on my mind:

• Using OpenAI() via vLLM has a chat_template already loaded on the serving side if one sends a list of messages with user/assistant roles.
• Using TGI directly via HFClientTGI. I am 100% sure this doesn't load any chat template. It's up to the user to implement the template before passing the prompt.
• Using OpenAI() via TGI since it now offers it similarly to vLLM. This also loads the chat_template and looks for a list of user/assistant messages.

One of my recent difficulties is logging/debugging how DSPy is actually sending the messages, how the prompt template looks, etc. I can't say if the template is respected for any of these three scenarios.