You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When constructing the requests for a MultipleChoiceTask (pre-refactored version), an implicit white space is added before the labels:
def construct_requests(self, doc, ctx):
lls = [
rf.loglikelihood(ctx, " {}".format(choice))[0] for choice in doc["choices"]
]
return lls
When using fine-tuned models for evaluation, which expect a specific format, the added white space causes the wrong prediction of label probabilities. How should I handle this for my custom fine-tuned llms?
The text was updated successfully, but these errors were encountered:
Hi,
When constructing the requests for a MultipleChoiceTask (pre-refactored version), an implicit white space is added before the labels:
When using fine-tuned models for evaluation, which expect a specific format, the added white space causes the wrong prediction of label probabilities. How should I handle this for my custom fine-tuned llms?
The text was updated successfully, but these errors were encountered: