fix: improve context relevancy metric #7964

davidsbatista · 2024-07-02T12:56:39Z

Related Issues

fixes #78

Proposed Changes:

The ContextRelevanceEvaluator now returns a list of relevant sentences for each context, instead of all the sentences in a context. Also, a score of 1 is now returned if a relevant sentence is found, and 0 otherwise.

How did you test it?

unit tests, integration tests, manual verification and end2end tests

Notes for the reviewer

Check the details of the discussion in #78 to better understand the context of this change.

Checklist

I have read the contributors guidelines and the code of conduct
I have updated the related issue with new insights and changes
I added unit tests and updated the docstrings
I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
I documented my code
I ran pre-commit hooks and fixed any issue

coveralls · 2024-07-02T13:04:08Z

Pull Request Test Coverage Report for Build 9761382356

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.001%) to 89.969%

Totals
Change from base Build 9758508178:	0.001%
Covered Lines:	6745
Relevant Lines:	7497

💛 - Coveralls

coveralls · 2024-07-02T15:16:46Z

Pull Request Test Coverage Report for Build 9763512100

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.003%) to 89.971%

Totals
Change from base Build 9758508178:	0.003%
Covered Lines:	6746
Relevant Lines:	7498

💛 - Coveralls

coveralls · 2024-07-02T15:46:00Z

Pull Request Test Coverage Report for Build 9763863648

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.003%) to 89.971%

Totals
Change from base Build 9758508178:	0.003%
Covered Lines:	6746
Relevant Lines:	7498

💛 - Coveralls

coveralls · 2024-07-02T16:30:43Z

Pull Request Test Coverage Report for Build 9764606380

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.003%) to 89.971%

Totals
Change from base Build 9758508178:	0.003%
Covered Lines:	6746
Relevant Lines:	7498

💛 - Coveralls

coveralls · 2024-07-02T17:19:29Z

Pull Request Test Coverage Report for Build 9764679440

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.003%) to 89.971%

Totals
Change from base Build 9758508178:	0.003%
Covered Lines:	6746
Relevant Lines:	7498

💛 - Coveralls

haystack/components/evaluators/context_relevance.py

coveralls · 2024-07-04T15:31:35Z

Pull Request Test Coverage Report for Build 9796828476

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0 changed or added relevant lines in 0 files are covered.
16 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.04%) to 90.003%

Files with Coverage Reduction	New Missed Lines	%
components/routers/conditional_router.py	2	97.4%
components/converters/azure.py	14	89.55%

Totals
Change from base Build 9758508178:	0.04%
Covered Lines:	6770
Relevant Lines:	7522

💛 - Coveralls

coveralls · 2024-07-05T10:45:21Z

Pull Request Test Coverage Report for Build 9807039259

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0 changed or added relevant lines in 0 files are covered.
17 unchanged lines in 3 files lost coverage.
Overall coverage increased (+0.04%) to 90.005%

Files with Coverage Reduction	New Missed Lines	%
components/builders/answer_builder.py	1	98.28%
components/routers/conditional_router.py	2	97.4%
components/converters/azure.py	14	89.55%

Totals
Change from base Build 9758508178:	0.04%
Covered Lines:	6772
Relevant Lines:	7524

💛 - Coveralls

coveralls · 2024-07-05T12:12:13Z

Pull Request Test Coverage Report for Build 9807972372

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0 changed or added relevant lines in 0 files are covered.
17 unchanged lines in 3 files lost coverage.
Overall coverage increased (+0.04%) to 90.004%

Files with Coverage Reduction	New Missed Lines	%
components/builders/answer_builder.py	1	98.28%
components/routers/conditional_router.py	2	97.4%
components/converters/azure.py	14	89.55%

Totals
Change from base Build 9758508178:	0.04%
Covered Lines:	6771
Relevant Lines:	7523

💛 - Coveralls

davidsbatista · 2024-07-08T07:58:46Z

NOTE: blocked until the 2.4 release

coveralls · 2024-07-11T13:11:01Z

Pull Request Test Coverage Report for Build 10041187329

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.003%) to 90.122%

Totals
Change from base Build 10005172669:	-0.003%
Covered Lines:	6861
Relevant Lines:	7613

💛 - Coveralls

haystack/components/evaluators/context_relevance.py

Co-authored-by: Madeesh Kannan <[email protected]>

davidsbatista · 2024-07-22T10:20:34Z

@shadeMe thanks for the feedback - I've updated the PR

shadeMe · 2024-07-22T10:40:48Z

Let's wait for @anakin87's review before merging.

haystack/components/evaluators/context_relevance.py

Co-authored-by: Stefano Fiorucci <[email protected]>

anakin87

Feel free to merge once the following line is updated

haystack/haystack/components/evaluators/context_relevance.py

Line 116 in ef1ef4b

"outputs" must be a dictionary with "statements" and "statement_scores".

davidsbatista added 4 commits July 2, 2024 14:10

fixing tests

68eab35

fixing tests

5c97fac

Merge branch 'main' into improve-context-relevancy-metric

f0d7484

updating tests

5ea4bda

github-actions bot added topic:tests type:documentation Improvements on the docs labels Jul 2, 2024

updating tests

3204739

updating docstring

40ea113

davidsbatista changed the title ~~Improve context relevancy metric~~ fix: improve context relevancy metric Jul 2, 2024

davidsbatista marked this pull request as ready for review July 2, 2024 16:24

davidsbatista requested a review from a team as a code owner July 2, 2024 16:24

davidsbatista requested review from Amnah199 and julian-risch and removed request for a team and Amnah199 July 2, 2024 16:24

adding release notes

a7b7b32

davidsbatista requested a review from a team as a code owner July 2, 2024 16:29

davidsbatista requested review from dfokina and removed request for a team July 2, 2024 16:29

davidsbatista added topic:eval 2.x Related to Haystack v2.0 labels Jul 2, 2024

davidsbatista requested a review from anakin87 July 3, 2024 08:59

mrm1001 reviewed Jul 4, 2024

View reviewed changes

haystack/components/evaluators/context_relevance.py Outdated Show resolved Hide resolved

making the insufficient information more robust

f238145

empty list instead of informative string

93b309f

shadeMe marked this pull request as draft July 8, 2024 08:33

shadeMe added this to the 2.4.0 milestone Jul 8, 2024

julian-risch removed their request for review July 10, 2024 16:51

Merge branch 'main' into improve-context-relevancy-metric

4facb67

davidsbatista added 2 commits July 12, 2024 10:48

Merge branch 'main' into improve-context-relevancy-metric

ed49292

Merge branch 'main' into improve-context-relevancy-metric

bf2181d

davidsbatista marked this pull request as ready for review July 19, 2024 11:13

shadeMe requested changes Jul 22, 2024

View reviewed changes

davidsbatista and others added 3 commits July 22, 2024 11:52

Update haystack/components/evaluators/context_relevance.py

af68087

Co-authored-by: Madeesh Kannan <[email protected]>

Update haystack/components/evaluators/context_relevance.py

f15eb0f

Co-authored-by: Madeesh Kannan <[email protected]>

fixing tests

9c23ceb

shadeMe approved these changes Jul 22, 2024

View reviewed changes

anakin87 requested changes Jul 22, 2024

View reviewed changes

haystack/components/evaluators/context_relevance.py Outdated Show resolved Hide resolved

davidsbatista and others added 3 commits July 22, 2024 13:19

Update haystack/components/evaluators/context_relevance.py

e366b27

Co-authored-by: Stefano Fiorucci <[email protected]>

reverting commit

64123b1

reverting again commit

ef1ef4b

anakin87 self-requested a review July 22, 2024 11:34

anakin87 approved these changes Jul 22, 2024

View reviewed changes

davidsbatista added 3 commits July 22, 2024 13:53

fixing docstrings

c358baf

removing deprecation warning

9c21997

removing warning import

308967c

davidsbatista merged commit 0c9dc00 into main Jul 22, 2024
17 checks passed

davidsbatista deleted the improve-context-relevancy-metric branch July 22, 2024 13:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: improve context relevancy metric #7964

fix: improve context relevancy metric #7964

davidsbatista commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 4, 2024 •

edited

Loading

coveralls commented Jul 5, 2024 •

edited

Loading

coveralls commented Jul 5, 2024 •

edited

Loading

davidsbatista commented Jul 8, 2024

coveralls commented Jul 11, 2024 •

edited

Loading

davidsbatista commented Jul 22, 2024

shadeMe commented Jul 22, 2024

anakin87 left a comment

fix: improve context relevancy metric #7964

fix: improve context relevancy metric #7964

Conversation

davidsbatista commented Jul 2, 2024 • edited Loading

Related Issues

Proposed Changes:

How did you test it?

Notes for the reviewer

Checklist

coveralls commented Jul 2, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9761382356

Details

💛 - Coveralls

coveralls commented Jul 2, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9763512100

Details

💛 - Coveralls

coveralls commented Jul 2, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9763863648

Details

💛 - Coveralls

coveralls commented Jul 2, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9764606380

Details

💛 - Coveralls

coveralls commented Jul 2, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9764679440

Details

💛 - Coveralls

coveralls commented Jul 4, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9796828476

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

coveralls commented Jul 5, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9807039259

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

coveralls commented Jul 5, 2024 • edited Loading

Pull Request Test Coverage Report for Build 9807972372

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

davidsbatista commented Jul 8, 2024

coveralls commented Jul 11, 2024 • edited Loading

Pull Request Test Coverage Report for Build 10041187329

Details

💛 - Coveralls

davidsbatista commented Jul 22, 2024

shadeMe commented Jul 22, 2024

anakin87 left a comment

Choose a reason for hiding this comment

davidsbatista commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 2, 2024 •

edited

Loading

coveralls commented Jul 4, 2024 •

edited

Loading

coveralls commented Jul 5, 2024 •

edited

Loading

coveralls commented Jul 5, 2024 •

edited

Loading

coveralls commented Jul 11, 2024 •

edited

Loading