Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create an example of how to improve a retriever on industry dataset #7439

Open
mrm1001 opened this issue Mar 28, 2024 · 0 comments
Open

create an example of how to improve a retriever on industry dataset #7439

mrm1001 opened this issue Mar 28, 2024 · 0 comments
Labels
P2 Medium priority, add to the next sprint if no P1 available topic:eval

Comments

@mrm1001
Copy link

mrm1001 commented Mar 28, 2024

Goal:
Showcase the Haystack evaluation metrics on an example that is close to what our users are trying to do, i.e. to improve a retriever in a RAG app.

The goal of this task is to find such an example, ideally finding:

  • the RAG pipeline architecture
  • the metrics to be used (if Haystack metrics are not finished yet, can use ragas/deepeval implementations for now). Here is the list of metrics that will be released in Haystack
  • the experiment to show how the retriever improves and the improvement should show in the metrics: ideally after tweaking chunk size, or changing embedding model/size.

This does not need to be a finished tutorial or even code, just need some scrap code somewhere (colab?).

Depends on: #7438

@mrm1001 mrm1001 added P1 High priority, add to the next sprint topic:eval labels Mar 28, 2024
@masci masci added P2 Medium priority, add to the next sprint if no P1 available and removed P1 High priority, add to the next sprint labels Mar 28, 2024
@masci masci changed the title find example of how to improve a retriever on industry dataset create an example of how to improve a retriever on industry dataset Apr 7, 2024
@mrm1001 mrm1001 assigned mrm1001 and TuanaCelik and unassigned mrm1001 and TuanaCelik Apr 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 Medium priority, add to the next sprint if no P1 available topic:eval
Projects
None yet
Development

No branches or pull requests

3 participants