Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add HaystackEvaluator #6904

Closed
tholor opened this issue Feb 5, 2024 · 1 comment
Closed

Add HaystackEvaluator #6904

tholor opened this issue Feb 5, 2024 · 1 comment
Assignees
Labels
2.x Related to Haystack v2.0 P2 Medium priority, add to the next sprint if no P1 available topic:eval

Comments

@tholor
Copy link
Member

tholor commented Feb 5, 2024

We are adding external integrations to enable eval with uptrain, deepeval and RAGAs.
While those come with many nice out-of-the box capabilities, we also see some limitations with them that motivates us to create an own Evaluator, natively in Haystack:

  1. Dependencies: deepeval and RAGAs come with several extra dependencies. We'd like to keep environments clean and small.
  2. Choose a different model: Most default to using GPT-4. We want to allow users to pick different providers. Especially for privacy relevant use cases, where only open source models are acceptable.
  3. Customization of model based metrics: Slightly lower prio than the other two but still important: users should be able to customize the prompt behind a metric (e.g. because slightly different wording is needed when they change an LLM or they want to provide different few-shot examples ...)
@tholor tholor added topic:eval P1 High priority, add to the next sprint 2.x Related to Haystack v2.0 labels Feb 5, 2024
@masci masci added P2 Medium priority, add to the next sprint if no P1 available and removed P1 High priority, add to the next sprint labels Feb 12, 2024
@masci
Copy link
Contributor

masci commented Feb 17, 2024

Closing in favor of #7022

@masci masci closed this as completed Feb 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.x Related to Haystack v2.0 P2 Medium priority, add to the next sprint if no P1 available topic:eval
Projects
None yet
Development

No branches or pull requests

5 participants