Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rethink testing strategy for HF-based components #7333

Open
anakin87 opened this issue Mar 7, 2024 · 0 comments
Open

Rethink testing strategy for HF-based components #7333

anakin87 opened this issue Mar 7, 2024 · 0 comments

Comments

@anakin87
Copy link
Member

anakin87 commented Mar 7, 2024

In Haystack 2.x, most components based on Hugging Face libraries don't have integration tests.

Why?
The integration tests were avoided mainly to keep the CI fast:

  • some of these tests would require downloading models and performing slow inference
  • others (for TGI and TEI) would require running docker services

The problem
We want the CI to be fast but integration tests are useful for detecting changes and bugs.
(We run some e2e tests that also cover some of these components, but I think they serve a different purpose and are not sufficient)

The plan
So I would propose that we rethink our strategy for testing these components.
I don't have strong ideas about that. Perhaps nightly integration tests?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant