Support Sparse Embedding Retrieval

[anakin87]

It is a feature the community asks for and is currently supported by Qdrant and Pinecone.

Update
I experimented with the complete round trip: from Document to sparse embedding stored in Qdrant/Pinecone and then querying (notebook).

What we need to do:

Tasks

Beta Give feedback

1. Investigate/design the integration
2. Introduce SparseEmbedding class and add it to Document
3. Make Document Stores initially skip SparseEmbedding haystack-core-integrations#604
   +0
   
4. release the SparseEmbedding class in 2.0.1
5. Introduce a first Sparse Embedder (feat(FastEmbed): Support for SPLADE Sparse Embedder haystack-core-integrations#579)
6. Make Qdrant write sparse embeddings (feat(Qdrant): start to work on sparse vector integration haystack-core-integrations#578)
7. Introduce Qdrant Sparse Embedding Retriever (feat(Qdrant): start to work on sparse vector integration haystack-core-integrations#578)
8. non-urgent: understand the problems related to Qdrant Hybrid Retriever
9. Pinecone: support hybrid retrieval haystack-core-integrations#695
   P3 feature request integration:pinecone +3
10. Sparse Embeddings support: create docs/material haystack-core-integrations#660
    3 of 3
    P1 integration:fastembed integration:qdrant type:documentation +4
    
11. Qdrant - add hybrid retriever haystack-core-integrations#675
    integration:qdrant type:documentation +2
12. The feature was announced through social media

[lambda-science]

Note:
As a 1st step: we now have working Sparse embedder in Haystack through FastEmbed integration
deepset-ai/haystack-core-integrations#579

[lambda-science]

Btw it would be cool to have a general BM25 Embedder in core haystack repo instead of relying only on Splade Embedder from FastEmbed :) As you already have a haystack-bm25

[lambda-science]

Note:
As a 2nd step: Qdrant integration could now support Sparse vector and be compatible with the FastEmbed sparse embedder from above 👀 deepset-ai/haystack-core-integrations#578