Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LLM-based Evaluators should return meta information provided by OpenAIGenerator #7905

Closed
bilgeyucel opened this issue Jun 20, 2024 · 1 comment · Fixed by #7947
Closed

LLM-based Evaluators should return meta information provided by OpenAIGenerator #7905

bilgeyucel opened this issue Jun 20, 2024 · 1 comment · Fixed by #7947
Assignees
Labels
2.x Related to Haystack v2.0 P1 High priority, add to the next sprint

Comments

@bilgeyucel
Copy link
Contributor

Is your feature request related to a problem? Please describe.
To calculate the cost of the evaluation, we need a token count. Most generator components in Haystack provide that information in the meta field.

Describe the solution you'd like
LLM-based evaluators should return token count information with the result. This would be handy for connecting evaluation pipelines to monitoring tools such as Langfuse

Describe alternatives you've considered
Leave it as it is

Additional context
N/A

@bilgeyucel bilgeyucel added the 2.x Related to Haystack v2.0 label Jun 20, 2024
@mrm1001 mrm1001 added the P1 High priority, add to the next sprint label Jun 21, 2024
@davidsbatista
Copy link
Contributor

davidsbatista commented Jun 28, 2024

It's ready for review only an issue with a bug in pylint:

- https://github.com/deepset-ai/haystack/actions/runs/9712238785/job/26806852617?pr=7947

Waiting for the new release of pylint (hopefully soon) to fix it.

FIXED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.x Related to Haystack v2.0 P1 High priority, add to the next sprint
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants