You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For generate_until output type tasks, I would like to be able to register metrics that are computed using the input prompt to the LLM as well as as the reference and prediction.
I have a niche use case for this but I think it would be useful more broadly. For example LLM-as-Judge style metrics would require the input prompt also.
Also happy to submit a PR if you can point me in the right direction.
The text was updated successfully, but these errors were encountered:
For generate_until output type tasks, I would like to be able to register metrics that are computed using the input prompt to the LLM as well as as the reference and prediction.
I have a niche use case for this but I think it would be useful more broadly. For example LLM-as-Judge style metrics would require the input prompt also.
Also happy to submit a PR if you can point me in the right direction.
The text was updated successfully, but these errors were encountered: