Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement abstract EvaluationHarness class #5

Merged
merged 7 commits into from
May 22, 2024

Conversation

shadeMe
Copy link
Contributor

@shadeMe shadeMe commented May 16, 2024

This is a part of a series of PRs that revolves around implementing an evaluation harness (c.f deepset-ai/haystack#7526).

EvaluationHarness provides the base interface for class that implement usecase-specific evaluation harnesses. It's meant to be generic over its inputs, overrides and outputs. A key feature of this abstraction is the support for overriding the initial parameters of arbitrary components in the evaluated (and evaluation) pipeline. It leverages the Pipeline abstraction's serialization hooks to transparently instantiate new pipeline internally for each set of overrides.

No unit tests are provided as this class does not implement any new functionality.

Blocked by #2.

@shadeMe shadeMe marked this pull request as ready for review May 22, 2024 09:28
@shadeMe shadeMe requested a review from a team as a code owner May 22, 2024 09:28
@shadeMe shadeMe requested review from vblagoje and removed request for a team May 22, 2024 09:28
@coveralls
Copy link

coveralls commented May 22, 2024

Pull Request Test Coverage Report for Build 9189156861

Details

  • 20 of 38 (52.63%) changed or added relevant lines in 2 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-6.0%) to 93.624%

Changes Missing Coverage Covered Lines Changed/Added Lines %
haystack_experimental/evaluation/eval_harness.py 18 36 50.0%
Totals Coverage Status
Change from base Build 9189123832: -6.0%
Covered Lines: 279
Relevant Lines: 298

💛 - Coveralls

@shadeMe shadeMe removed the request for review from vblagoje May 22, 2024 09:57
@shadeMe shadeMe merged commit 57ece86 into deepset-ai:main May 22, 2024
5 checks passed
@shadeMe shadeMe deleted the feat/eval-harness-base-class branch May 22, 2024 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants