You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
For evaluating and optimizing generative pipelines it's essential to know which prompt (incl. full context) was used for each result. E.g. was the relevant document part of the prompt?
Currently we only have this info for PromptNode in the debug output per pipeline run. However we are lacking it on the result (i.e. Answer) level. Additionally OpenAIAnswerGenerator doesn't even provide this information. Pipeline.eval does not store the PromptNode prompt, so any analysis of the EvaluationResult is lacking the prompt.
Describe the solution you'd like
store prompt for PromptNode and OpenAIAnswerGenerator in Answer objects (e.g. meta or dedicated field)
store prompt to EvaluationResult dataframes
Describe alternatives you've considered
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
For evaluating and optimizing generative pipelines it's essential to know which prompt (incl. full context) was used for each result. E.g. was the relevant document part of the prompt?
Currently we only have this info for PromptNode in the debug output per pipeline run. However we are lacking it on the result (i.e. Answer) level. Additionally OpenAIAnswerGenerator doesn't even provide this information. Pipeline.eval does not store the
PromptNode
prompt, so any analysis of theEvaluationResult
is lacking the prompt.Describe the solution you'd like
PromptNode
andOpenAIAnswerGenerator
inAnswer
objects (e.g. meta or dedicated field)EvaluationResult
dataframesDescribe alternatives you've considered
Additional context
Add any other context or screenshots about the feature request here.
The text was updated successfully, but these errors were encountered: