Added cookbook for ragas integration #70
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
Hi @TuanaCelik! Would love your feedback on my PR. It fixes #52. |
| @@ -0,0 +1,5272 @@ | |||
| { | |||
There was a problem hiding this comment.
| @@ -0,0 +1,5272 @@ | |||
| { | |||
There was a problem hiding this comment.
Line #2.
nitpick: Let's do pip install as other notebooks, one liner with ! i.e.
!pip install "pydantic<1.10.10", datasets>=2.6.1", haystack-ai, ragas-haystack
Reply via ReviewNB
| @@ -0,0 +1,5272 @@ | |||
| { | |||
There was a problem hiding this comment.
Please add a short note here that we are using SQUAD here for convenience and tutorial purposes and that users will likely eval their domain specific datasets.
Also note what are the annotation requirements needed in a dataset, i.e which datasets fields are what you described further below with "Now that we have the questions, generated answers, contexts and the ground truths, we can begin our pipeline evaluation and compute all the supported metrics."
It is prudent to notify users of these requirements before they proceed further.
Reply via ReviewNB
| @@ -0,0 +1,5272 @@ | |||
| { | |||
There was a problem hiding this comment.
Line #24. os.environ["OPENAI_API_KEY"] = getpass("Enter OpenAI API key:")
No need to do this again, you've already completed intake of the OPENAI_API_KEY above. So either here or there, either is fine, but not both.
Reply via ReviewNB
| @@ -0,0 +1,5272 @@ | |||
| { | |||
There was a problem hiding this comment.
Let's have a nice metric definition like you do in other sections :-)
Context Utilization is....
Reply via ReviewNB
| @@ -0,0 +1,5272 @@ | |||
| { | |||
There was a problem hiding this comment.
There was a problem hiding this comment.
Updated the metric description with the definition
| @@ -0,0 +1,5272 @@ | |||
| { | |||
There was a problem hiding this comment.
| @@ -0,0 +1,5272 @@ | |||
| { | |||
There was a problem hiding this comment.
| @@ -0,0 +1,5272 @@ | |||
| { | |||
There was a problem hiding this comment.
| @@ -0,0 +1,5272 @@ | |||
| { | |||
There was a problem hiding this comment.
Why not do all at once - for example all metrics that don't require ground truth or need common fields?
Reply via ReviewNB
There was a problem hiding this comment.
I'm not entirely clear on your question. Could you provide more details?
I included this section to demonstrate the computation of multiple metrics using a single pipeline object.
There was a problem hiding this comment.
@AnushreeBannadabhavi thanks a lot for this effort and apologies for delayed review. Although there are some minor request for changes this is already a solid work and an amazing contribution. Let's polish up some of the details and integrate this one 🚀
|
@AnushreeBannadabhavi as we are, unfortunately, time bound for the integration of this PR would you please make these updates until tomorrow (12.4.24)? If not, I'll make them myself and we'll integrate this PR. Let us know. |
|
Hi @vblagoje! Thank you for your review. I was busy yesterday and hence I wasn't able to address review comments. |
|
Awesome, thank you @AnushreeBannadabhavi |
|
Hi @vblagoje! I've updated the notebook and addressed all your review comments except the last one (multiple evaluator components). I can take a look at that quickly once I hear back from you. |
|
Thanks a lot @AnushreeBannadabhavi for a quick turnaround! @TuanaCelik the notebook, looks ok to me. I'll let you have the final word and integrate |
|
@davidsbatista this one is very similar to DeepEval, would you please also have a look? |
|
@vblagoje LGTM, approved. I was confused because all the "changes" in the review were still "open" - I think it's better to hit the button "mark as solved" if it's already solved. |
A cookbook showcasing the Haystack-Ragas integration. Fixes the issue Cookbook for Ragas