Cannot use OpenAIGenerator or HuggingFaceLocalGenerator on Databricks - package conflict?

[lukasz-kastelik]

Describe the bug
While importing
from haystack.components.generators import OpenAIGenerator or from haystack.components.generators import HuggingFaceLocalGenerator on Databricks Runtime 14.3 LTS or 15.1 ML you get an error:
Error message
Error that was thrown (if available)

Expected behavior
cannot import name 'Iterator' from 'typing_extensions' (/databricks/python/lib/python3.10/site-packages/typing_extensions.py)

Additional context
Full list of Python dependencies available in Databricks is listed here: https://docs.databricks.com/en/release-notes/runtime/14.3lts.html

To Reproduce
Provision a Databricks cluster and install the newest verstion of haystack-ai using pip.

FAQ Check

• Have you had a look at our new FAQ page?

[anakin87]

Hey!

The issue seems similar to this stack overflow thread.

Which version of typing-extension is installed?
Can you upgrade it to typing-extension>=4.7?

(probably a duplicate of #7647)

[lukasz-kastelik]

Hey!

The issue seems similar to this stack overflow thread.

Which version of typing-extension is installed? Can you upgrade it to typing-extension>=4.7?

(probably a duplicate of #7647)

I tried
%pip install --force-reinstall typing-extensions>=4.7
with no effect

[anakin87]

Has the new version of typing-extensions been correctly installed? Please check the output of ! pip freeze|grep typing...

If yes, can you force reinstalling Haystack and restarting the kernel (IDK if this is feasible on databricks)?