HuggingFace Embedder potential deadlock

[srini047]

Describe the bug
I was running the following query pipeline:

query = Pipeline()
query.add_component("embedder", SentenceTransformersTextEmbedder())
query.add_component("retriever", QdrantEmbeddingRetriever(QdrantDocumentStore()))
query.add_component("prompt_builder", PromptBuilder(template=template))
query.add_component(
    "generator",
    OpenAIGenerator(api_key=Secret.from_token()),
)

# query.connect("embedder.embedding", "retriever.query_embedding", "prompt_builder.query_embedding")
query.connect("embedder.embedding", "retriever.query_embedding")
query.connect("retriever", "prompt_builder.documents")
query.connect("prompt_builder", "generator")

Warning message

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Relevant stackoverflow thread.

Expected behavior
This warning should not be thrown as this can cause the system to hang and consume entire resources in the worst case.

Additional context
Better to disable TOKENIZERS_PARALLELISM by default so the GIL logic doesn't invoke deadlock.

To Reproduce
Run the query pipeline provided above.

FAQ Check

• Have you had a look at our new FAQ page?

System:

• OS: MacOS
• GPU/CPU:
• Haystack version (commit or version number): 2.1.2
• DocumentStore: QdrantDocumentStore
• Reader: -
• Retriever: QdrantEmbeddingRetriever

[anakin87]

Hello!

I tried to understand the warning and I found this explanation from the Hugging Face team:

This is happening whenever you use multiprocessing (Often used by data loaders). The way to disable this warning is to set the TOKENIZERS_PARALLELISM environment variable to the value that makes more sense for you. By default, we disable the parallelism to avoid any hidden deadlock that would be hard to debug, but you might be totally fine while keeping it enabled in your specific use-case.
You can try to set it to true, and if your process seems to be stuck, doing nothing, then you should use false.

So it seems that when you see this warning message, tokenizers parallelism has been automatically disabled to avoid deadlocks.
Based on my understanding, Haystack users have no risk of deadlock. However, if a user wants to use tokenizers parallelism, they can set the corresponding environment variable.