[data] Windows Python 3.11 cannot install due to pyarrow version pin

[nrudakov]

What happened + What you expected to happen

pip install ray[tune] exits with an error when running on Windows Python 3.11.

Collecting ray[tune]
  Obtaining dependency information for ray[tune] from https://files.pythonhosted.org/packages/54/f4/23256815e8d33ddedc877d2093e63dc97dae6b37742a2e4d00ca0ba9aeb2/ray-2.6.1-cp311-cp311-win_amd64.whl.metadata
  Using cached ray-2.6.1-cp311-cp311-win_amd64.whl.metadata (12 kB)
Collecting click>=7.0 (from ray[tune])
  Obtaining dependency information for click>=7.0 from https://files.pythonhosted.org/packages/1a/70/e63223f8116931d365993d4a6b7ef653a4d920b41d03de7c59499962821f/click-8.1.6-py3-none-any.whl.metadata
  Using cached click-8.1.6-py3-none-any.whl.metadata (3.0 kB)
Collecting filelock (from ray[tune])
  Obtaining dependency information for filelock from https://files.pythonhosted.org/packages/00/45/ec3407adf6f6b5bf867a4462b2b0af27597a26bd3cd6e2534cb6ab029938/filelock-3.12.2-py3-none-any.whl.metadata
  Using cached filelock-3.12.2-py3-none-any.whl.metadata (2.7 kB)
Collecting jsonschema (from ray[tune])
  Obtaining dependency information for jsonschema from https://files.pythonhosted.org/packages/2b/ff/af59fd34bc4d7ac3e6e0cd1f3c10317d329b6c1aee179e8b24ad9a79fbac/jsonschema-4.19.0-py3-none-any.whl.metadata
  Using cached jsonschema-4.19.0-py3-none-any.whl.metadata (8.2 kB)
Collecting msgpack<2.0.0,>=1.0.0 (from ray[tune])
  Using cached msgpack-1.0.5-cp311-cp311-win_amd64.whl (60 kB)
Collecting packaging (from ray[tune])
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting protobuf!=3.19.5,>=3.15.3 (from ray[tune])
  Obtaining dependency information for protobuf!=3.19.5,>=3.15.3 from https://files.pythonhosted.org/packages/1c/1f/e64ec8bc540266178d4d0e0255e0c1427ba284ba951e193d193a238b09bb/protobuf-4.24.0-cp310-abi3-win_amd64.whl.metadata
  Using cached protobuf-4.24.0-cp310-abi3-win_amd64.whl.metadata (540 bytes)
Collecting pyyaml (from ray[tune])
  Obtaining dependency information for pyyaml from https://files.pythonhosted.org/packages/b3/34/65bb4b2d7908044963ebf614fe0fdb080773fc7030d7e39c8d3eddcd4257/PyYAML-6.0.1-cp311-cp311-win_amd64.whl.metadata
  Using cached PyYAML-6.0.1-cp311-cp311-win_amd64.whl.metadata (2.1 kB)
Collecting aiosignal (from ray[tune])
  Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting frozenlist (from ray[tune])
  Obtaining dependency information for frozenlist from https://files.pythonhosted.org/packages/39/16/72d9ccd30815d0b37218348f053be37bc3d14288ac192a794a39990ac28e/frozenlist-1.4.0-cp311-cp311-win_amd64.whl.metadata
  Using cached frozenlist-1.4.0-cp311-cp311-win_amd64.whl.metadata (5.3 kB)
Collecting requests (from ray[tune])
  Obtaining dependency information for requests from https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
  Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting grpcio>=1.42.0 (from ray[tune])
  Obtaining dependency information for grpcio>=1.42.0 from https://files.pythonhosted.org/packages/dd/10/26829ba71579a0248ad32211c2964fc389800ec58a9de8822b9a733cd800/grpcio-1.56.2-cp311-cp311-win_amd64.whl.metadata
  Using cached grpcio-1.56.2-cp311-cp311-win_amd64.whl.metadata (4.1 kB)
Collecting numpy>=1.19.3 (from ray[tune])
  Obtaining dependency information for numpy>=1.19.3 from https://files.pythonhosted.org/packages/72/b2/02770e60c4e2f7e158d923ab0dea4e9f146a2dbf267fec6d8dc61d475689/numpy-1.25.2-cp311-cp311-win_amd64.whl.metadata
  Using cached numpy-1.25.2-cp311-cp311-win_amd64.whl.metadata (5.7 kB)
Collecting pandas (from ray[tune])
  Obtaining dependency information for pandas from https://files.pythonhosted.org/packages/9e/71/756a1be6bee0209d8c0d8c5e3b9fc72c00373f384a4017095ec404aec3ad/pandas-2.0.3-cp311-cp311-win_amd64.whl.metadata
  Using cached pandas-2.0.3-cp311-cp311-win_amd64.whl.metadata (18 kB)
Collecting tensorboardX>=1.9 (from ray[tune])
  Obtaining dependency information for tensorboardX>=1.9 from https://files.pythonhosted.org/packages/44/7b/eee50dcadcee4c674353ca207fdcd53a5b1f382021af1ed1797f9c0c45d2/tensorboardX-2.6.2-py2.py3-none-any.whl.metadata
  Using cached tensorboardX-2.6.2-py2.py3-none-any.whl.metadata (5.7 kB)
Collecting pyarrow<7.0.0,>=6.0.1 (from ray[tune])
  Using cached pyarrow-6.0.1.tar.gz (770 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [9 lines of output]
      Ignoring numpy: markers 'python_version < "3.8"' don't match your environment
      Ignoring numpy: markers 'python_version == "3.8"' don't match your environment
      Ignoring numpy: markers 'python_version == "3.9"' don't match your environment
      Collecting cython>=0.29
        Obtaining dependency information for cython>=0.29 from https://files.pythonhosted.org/packages/7d/61/bf165c17a1296fd7db78e18fd8cbb157ab04060ec58d34ff319424af3e2d/Cython-3.0.0-cp311-cp311-win_amd64.whl.metadata
        Using cached Cython-3.0.0-cp311-cp311-win_amd64.whl.metadata (3.2 kB)
      ERROR: Ignored the following versions that require a different python version: 1.21.2 Requires-Python >=3.7,<3.11; 1.21.3 Requires-Python >=3.7,<3.11; 1.21.4 Requires-Python >=3.7,<3.11; 1.21.5 Requires-Python >=3.7,<3.11; 1.21.6 Requires-Python >=3.7,<3.11
      ERROR: Could not find a version that satisfies the requirement numpy==1.21.3 (from versions: 1.3.0, 1.4.1, 1.5.0, 1.5.1, 1.6.0, 1.6.1, 1.6.2, 1.7.0, 1.7.1, 1.7.2, 1.8.0, 1.8.1, 1.8.2, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.10.0.post2, 1.10.1, 1.10.2, 1.10.4, 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.12.0, 1.12.1, 1.13.0, 1.13.1, 1.13.3, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.15.4, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4, 1.16.5, 1.16.6, 1.17.0, 1.17.1, 1.17.2, 1.17.3, 1.17.4, 1.17.5, 1.18.0, 1.18.1, 1.18.2, 1.18.3, 1.18.4, 1.18.5, 1.19.0, 1.19.1, 1.19.2, 1.19.3, 1.19.4, 1.19.5, 1.20.0, 1.20.1, 1.20.2, 1.20.3, 1.21.0, 1.21.1, 1.22.0, 1.22.1, 1.22.2, 1.22.3, 1.22.4, 1.23.0rc1, 1.23.0rc2, 1.23.0rc3, 1.23.0, 1.23.1, 1.23.2, 1.23.3, 1.23.4, 1.23.5, 1.24.0rc1, 1.24.0rc2, 1.24.0, 1.24.1, 1.24.2, 1.24.3, 1.24.4, 1.25.0rc1, 1.25.0, 1.25.1, 1.25.2)
      ERROR: No matching distribution found for numpy==1.21.3
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Installation succeeds on Python 3.10.

Versions / Dependencies

ray[tune] 2.6.1 2.6.2, Python 3.11.4, Windows 11 Business 22H2 22621.2134, pip 23.2.1, setuptools 68.0.0

Reproduction script

pip install ray[tune]

Issue Severity

High: It blocks me from completing my task.

[RocketRider]

I get the same error, would love to update to Python 3.11.
Tested with ray 2.6.3.

[nrudakov]

Same problem with 2.7.0 and 2.7.1

[matthewdeng]

@c21 I think this is technically due to a requirement from Ray Data?

ray/python/setup.py

Lines 232 to 236 in 2104d63

	if sys.platform != "win32":
	pyarrow_dep = "pyarrow >= 6.0.1"
	else:
	# Serialization workaround for pyarrow 7.0.0+ doesn't work for Windows.
	pyarrow_dep = "pyarrow >= 6.0.1, < 7.0.0"

A few options:

1. If this is fixed in Arrow already we can remove the custom serialization logic and remove this limit.
2. Another option if this is specific to Ray Data is to change the upper bound to only apply to Ray Data.

[c21]

Discussed with @matthewdeng offline, we are thinking we can probably make the version check optional. The only risk you running PyArrow 7+ on Windows, is that you will have significant performance regression when using Ray Data. But if your workload is not using Ray Data, it's fine to ignore.

[scottjlee]

The action items to be addressed are:

• 1. Remove the hard requirement check for Windows pyarrow < 7
• 2. Add runtime check for Pyarrow version >= 7 in Dataset.__init__() (or other TBD appropriate code path)
• 3. Handle any remaining behavior gap related to _register_arrow_data_serializer(), and include an environment variable to override the Pyarrow version check

[amotl]

Hi there,

thanks for your work on this. We just wanted to report that this error is also tripping PyCaret's CI when attempting to add Python 3.11 to the test matrix.

With kind regards,
Andreas.

/cc @Yard1

References

• Installation fails on Python 3.11/Windows tune-sklearn#283
• More Python 3.11 support pycaret/pycaret#3814 (comment)
• Optimize CI pycaret/pycaret#3815 (comment)

[anyscalesam]

@mattip followed up with @scottjlee and assigned to you; can you please take up the action item that Scott laid out two comments up?

[mattip]

Closing since #42097 was merged. Please open a new issue if this causes performance problems