Suboptimal Performance on Generation Tasks

[erenup]

Hi @haileyschoelkopf

I find one general problem for generation tasks when I am developing the selfcheckgpt task.

The problem is that:

1. For each task, we can set one prompt format in task.yaml. But different models have their own system prompts or starting token. So one task can not be compared across multiple models.
2. For each task, we can set stopping tokens in the {"until": ["\n\n"}. But different models have their own ending tokens. So the model will generate something like this:
   

I think this is a general problem and more people who are using this framework should be aware of it when they are trying to compare different models in on generation task.

Thank you very much.

[StellaAthena]

We are currently working on supporting HF Chat Templating, see #1287

[haileyschoelkopf]

Thanks @erenup for the feedback! With respect to the latter point--we currently support the passing of generation hyperparameters via the --gen_kwargs CLI flag. I can see about adding the ability to provide extra stop sequences via the CLI using this flag.

[erenup]

Thank you very much for the quick reply!

[erenup]

hi @haileyschoelkopf

I think this could be a potential way to fix the huggingface.py model generation performance with apply_chat_template:
add self.tokenizer.apply_chat_template(messages, tokenize=False) to tok_batch_encode function in huggingface.py

How do you think about it?

[LSinev]

Is this still an issue after #1873?