[Core] Unable to install ray[all] in python

[grizzlybearg]

What happened + What you expected to happen

I realized that one of the requirements blocking the installation of ray in python 3.11 is pyarrow<7.0.0,>=6.0.1 (from ray[tune]) which is wierd since the compatible version of pyarrow (6.0.1) was released on November 18 2021. This makes it incompatible with many packages in Q3 2023. Is it possible to bump this requirement higher closer to pyarrow 11 or 12?

Versions / Dependencies

Python 3.11

Reproduction script

pip install -U "ray[all]"

Issue Severity

High: It blocks me from completing my task.

[RocketRider]

This blocks me as well. The issue seems to be only related to windows. On Linux I can install ray 2.7 in python 3.11.

[rkooo567]

I think it is the data package (not core). cc @c21 can you take a look?

[rkooo567]

cc @c21. This seems like a blocker for supporting python 3.11.

[scottjlee]

Seems that this is not related to python 3.11, but the restriction comes from using Windows with Pyarrow 7+ (see setup.py). It looks like our internal fix for the Arrow bug, implemented in #29993, is not compatible with Windows.

[anyscalesam]

@richardliaw can you advise on next steps here?

[AvisP]

Having the same issue, so how to solve it? As mentioned here #39222 for installing rllib I don't need it but getting the error related to arrow when doing a pip install "ray[rllib]" Is there a way around?

[AvisP]

So I found a temporary workaround if you are using ray such that is not using the pyarrow package.

First I uninstalled existing ray installation. Then I got the source code for pyarrow 6.0.1 from here. In the package I went to ./python and changed the version dependencies numpy==1.21.3 to numpy>=1.21.3 in the files requirements-wheel-build.txt, requirements-wheel-test.txt and also in the setup.py file changed to install_requires = ('numpy >= 1.23.1',) Next I installed cmake and Visual Studio Community version. I was unsure about the individual components that is needed to install for Visual Studio and just get the usual CRT and compilers. Once all that is done did a python setup.py install after navigating to the directory and it installed without further errors! However when I started python and did a import pyarrow it failed with a message

"....\_init__.py", line 63 in <module> import pyarrow.lib as _lib ModuleNotFoundError: NO module named "pyarrow.lib"

Now that pyarrow 6.0.1 is installed, when I did a pip install -U "ray[rllib]" it installed without issues and able to use it to run my script. I am not knowledgable with python packages but maybe someone else can provide a better solution that builds the pyarrow package properly.

[jorisvandenbossche]

Seems that this is not related to python 3.11, but the restriction comes from using Windows with Pyarrow 7+ (see setup.py). It looks like our internal fix for the Arrow bug, implemented in #29993, is not compatible with Windows.

Speaking as a pyarrow maintainer here: I personally doubt that this is the case. I would suggest that you actually test this again on CI by removing the upper pin and do a Windows test run with the latest pyarrow to check if this indeed still the case, or whether it's fine to remove this pin.

Nothing in that original PR for fixing the serialization issue looks as something that might have issues on Windows. Of course, there might have been a Windows-specific bug in pyarrow 7 in the IPC serialization that surfaced by switching to that. But 1) it might be that this is already fixed in the meantime for more recent pyarrow versions, and 2) if there is actually a bug for Windows, it would be nice to report that to pyarrow so we could actually try to fix that.

[raulchen]

@jorisvandenbossche thanks, we'll try removing the upper pin first.

[anyscalesam]

@scottjlee can you confirm if dupe; than close

[scottjlee]

confirming that this is a dupe of #38300; since that issue was created first and has other discussion, we will move to discuss the topic there.