Migrate to numpy 2.0

[cclauss]

Feature description

The beta release of numpy is now available so we should run the appropriate ruff command and ensure our algorithms are compatible... https://numpy.org/devdocs/numpy_2_0_migration_guide.html

@MaximSmolskiy

[MaximSmolskiy]

@cclauss I think this rule is already checked and satisfied.

Command

ruff check --select NPY201

does not show any violations.

And in https://docs.astral.sh/ruff/rules/#numpy-specific-rules-npy I do not see any specific signs near NPY201 rule to believe that it is not included by default in https://github.com/TheAlgorithms/Python/blob/master/pyproject.toml#L48

[cclauss]

Are there other tasks required to complete the upgrade?

[MaximSmolskiy]

As I understand, experiment in #11340 shows that some packages (statsmodels, scikit-learn and maybe some more) are not ready for numpy 2.0 - there are problems on Install dependencies stage

[ramilili]

I truly wish you a good luck

[quant12345]

In which folder were the problems or show how to run all tests? I want to see what the problems are. In June I did a PR in statsmodels, they had one build numpy version --- 2.1.0.dev0 and there were no failures. I don't know about the current time.

[cclauss]

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip setuptools six wheel
python -m pip install pytest-cov -r requirements.txt
pytest

From .github/workflows/build.yml

[quant12345]

After installation: pip install 'numpy==2.0.1' --force-reinstall

I receive the following warnings:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
imageio 2.35.0 requires numpy<2.0.0, but you have numpy 2.0.1 which is incompatible.
tensorflow 2.17.0 requires numpy<2.0.0,>=1.26.0; python_version >= "3.12", but you have numpy 2.0.1 which is incompatible.

There are a few files where the tests fail, but there are no errors, and the following is printed:

platform linux -- Python 3.12.3, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/quant/PycharmProjects/algorithms
configfile: pyproject.toml
plugins: cov-5.0.0
collecting ... Fatal Python error: Illegal instruction

Current thread 0x000078081f9e1080 (most recent call first):
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1289 in create_module
  File "<frozen importlib._bootstrap>", line 813 in module_from_spec
  File "<frozen importlib._bootstrap>", line 921 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1415 in _handle_fromlist
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/tensorflow/python/platform/self_check.py", line 63 in preload_check
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/tensorflow/python/pywrap_tensorflow.py", line 34 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 995 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1415 in _handle_fromlist
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/tensorflow/__init__.py", line 38 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 995 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "/home/quant/PycharmProjects/algorithms/computer_vision/cnn_classification.py", line 27 in <module>
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/assertion/rewrite.py", line 174 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1387 in _gcd_import
  File "/usr/lib/python3.12/importlib/__init__.py", line 90 in import_module
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/pathlib.py", line 582 in import_path
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/python.py", line 493 in importtestmodule
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/python.py", line 546 in _getobj
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/python.py", line 284 in obj
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/doctest.py", line 565 in collect
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 389 in collect
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 391 in pytest_make_collect_report
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 567 in collect_one_node
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/main.py", line 835 in _collect_one_node
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/main.py", line 970 in genitems
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/main.py", line 809 in perform_collect
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/main.py", line 347 in pytest_collection
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/main.py", line 336 in _main
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/main.py", line 283 in wrap_session
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/main.py", line 330 in pytest_cmdline_main
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/home/quant/PycharmProjects/algorithms/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/home/quant/PycharmProjects/algorithms/.venv/bin/pytest", line 8 in <module>

Extension modules: numpy._core._multiarray_umath, numpy._core._multiarray_tests, numpy.linalg._umath_linalg (total: 3)

Invalid instruction (memory image flushed to disk)

Files in which tests do not run(numpy 2.0.1):
computer_vision/cnn_classification.py, dynamic_programming/k_means_clustering_tensorflow.py, machine_learning/lstm, machine_learning/sequential_minimum_optimization.py, neural_network/input_data.py.

Usually there is tensorflow there(I'm not exactly sure, but I'll assume that the reason is tensorflow).
There's something on the topic here.

And there are many test failures where numbers do not match when printed in the new numpy (the number is printed with the type).

Example: pytest electronics/circular_convolution.py

FAILURES:

Expected:
    [10, 10, 6, 14]
Got:
    [np.int64(10), np.int64(10), np.int64(6), np.int64(14)]

I can fix it (this does not apply to the five files I listed above):

1. Either convert numbers, arrays to Python instead of numpy.
2. Fix the tests themselves: 12.0 by replacing with np.float64(12.0)
3. You can put np.set_printoptions(legacy="1.25") .

Thoughts?

[cclauss]

My vote would be 1. convert numbers, arrays to Python instead of numpy.

[int(x) for x in y]

[cclauss]

v2.1.0 was just released https://pypi.org/project/numpy/#history

[quant12345]

Unable to launch: 'Getting requirements to build wheel did not run successfully'. I see a line with statsmodels. Although in their requirements.txt: numpy>=1.22.3,<3 .

I tried to return numpy-1.26.4 to requirements.txt, it works. In general, it is not clear yet what to do.?

Update 20.08.2024

So I found the ecosystem numpy 2.0 table. Judging by it, keras and tensorflow are not marked in it.
Also not: beautifulsoup4, fake_useragent, lxml, pillow, qiskit, qiskit-aer, requests, rich, tweepy, typing_extensions.

Last thing I tried: numpy>=2.0.0, matplotlib>=3.8.4, opencv-python>=4.10.0.84, pandas>=2.2.2, scikit-learn>=1.4.2, statsmodels>=0.14.2. Which didn't work.