You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We are adding external integrations to enable eval with uptrain, deepeval and RAGAs.
While those come with many nice out-of-the box capabilities, we also see some limitations with them that motivates us to create an own Evaluator, natively in Haystack:
Dependencies: deepeval and RAGAs come with several extra dependencies. We'd like to keep environments clean and small.
Choose a different model: Most default to using GPT-4. We want to allow users to pick different providers. Especially for privacy relevant use cases, where only open source models are acceptable.
Customization of model based metrics: Slightly lower prio than the other two but still important: users should be able to customize the prompt behind a metric (e.g. because slightly different wording is needed when they change an LLM or they want to provide different few-shot examples ...)
The text was updated successfully, but these errors were encountered:
We are adding external integrations to enable eval with uptrain, deepeval and RAGAs.
While those come with many nice out-of-the box capabilities, we also see some limitations with them that motivates us to create an own Evaluator, natively in Haystack:
The text was updated successfully, but these errors were encountered: