feat: Add filter_policy to pgvector integration

[vblagoje]

Why:

Adds flexible filtering options to pgvector integration. By introducing a filter policy option (replace or merge), developers can now control how runtime filters are applied relative to initialization time filters.

• partially fixes Add filter_policy init parameter to all retrievers #780
• supersedes feat: Add filter_policy init parameter to all retrievers #781

What:

• Added filter_policy parameter with options replace and merge across multiple retrievers to control filter behavior dynamically.
• Unit tests were updated or added to validate the new functionality.

How can it be used:

• Dynamic Filter Behavior Adjustment: Users can decide whether to completely override the initial filters set during the retriever's initialization (replace) or merge them with runtime filters, with the latter taking precedence (merge).

• Complex Search Scenarios:

  • In cases where the context of a query might dictate altering pre-set filters without discarding them, the merge option allows for an additive approach.
  • For strict query contexts that require ignoring initial filters, the replace option offers a clean slate for filters at runtime.

How did you test it:

• Unit tests were enhanced or newly created to cover both replace and merge scenarios for the filter_policy parameter.
• Tests ensure that filter logic is correctly applied based on the policy setting, whether it merges runtime filters with initial filters or replaces them entirely.
• Additional test cases should be considered for complex filter merge scenarios to ensure priority and override mechanisms function as expected.

[anakin87]

In the docstrings of the run method, I would mention that the way runtime filters are applied depends on the filter_policy chosen at document store initialization.

[vblagoje]

Good feedback, thanks @anakin87 , will apply across the board

[anakin87]

Could you also add a test as requested/done in #823 (comment)?

[vblagoje]

Could you also add a test as requested/done in #823 (comment)?

Yes, already on it - for all PRs 😓

[anakin87]

Could you also add a test as requested/done in #823 (comment)?

Yes, already on it - for all PRs 😓

I understand your pain.
Perhaps you could create a utility function to handle filters in the Haystack repository and test it only there once and for all.

[vblagoje]

@anakin87 we need to use the already released https://github.com/deepset-ai/haystack/blob/main/haystack/document_stores/types/filter_policy.py instead of literal, across the board as well. I'll add that change here.

[vblagoje]

Please delay review until a new patch release of 2.2.x is released. We need these two PRs to be included in a patch release first.

[vblagoje]

Ready for review now @anakin87

• Since you last looked at this PR we introduced apply_filter_policy function that is used by all Retrievers in this PR and we thoroughly tested that function in that PR. We then just applied it here in run method
• Please note if FilterPolicy type is used consistently both in init and in run, and that it is documented well

[vblagoje]

Stefano recommended one more small improvement. Please hold your review until a new commit with it is added here

[anakin87]

please incorporate the suggestions and merge 🙂

[vblagoje]

@anakin87 one last look 🙏

[anakin87]

looks good