BLD: build with numpy instead of oldest-supported-numpy #478
Conversation
|
I also updated the minimal requirement for numpy to sync it with |
6d5a94e
to
49b8e33
Compare
|
Your suggestion sounds good to me. However, there are still some issues in building the wheels in CI: |
|
Oh, I didn't see this when I tested locally, but luckilly I already had to deal with this exact change in a different package downstream of NumPy ! I should be able to fix this quickly. |
|
I was able to reproduce the crash locally when I switched from Python 3.12 to 3.9, and I just pushed a fix. |
|
LGTM. Thanks @neutrinoceros ! |
|
This was included in numexpr 2.10, right? However I see an import failure of numexpr claiming it wasn't compiled against numpy 2.0 here: Any idea? Or am I misreadig this and it's not actually numexpr? |
|
It was indeed included in 2.10 so I think a different dependency may be adding constraints which prevent 2.10 from being installed. I would recommend running |
|
Thanks! I saw 2.10 but that was only in the isolated wheel build environment, indeed in the actual runtime environment, the old version was installed. |
There was a problem hiding this comment.
Thanks Clément and Francesc! 🙏
It is nice to see the NumPy 2 upgrade in numexpr was relatively straightforward 🥳
Have noted a couple things we might be able to smooth out, which should also make maintenance easier going forward
cc @seberg (for awareness)
| def_macros = [ | ||
| # keep in sync with minimal runtime requirement (requirements.txt) | ||
| ('NPY_TARGET_VERSION', 'NPY_1_19_API_VERSION') | ||
| ] |
There was a problem hiding this comment.
AIUI this shouldn't be needed unless numexpr is doing something more particular with the NumPy API
During the conda-forge NumPy 2 bringup discussion we talked to @rgommers about this, Ralf wrote up a nice explanation here ( conda-forge/conda-forge.github.io#1997 (comment) ). Quoting that below:
- numpy's
pyproject.tomlcontains and enforces the lowest-supported version, e.g.:requires-python = ">=3.9"- the first numpy version to support that python version (e.g.,
1.19.0for py39) determined the maximum version thatNPY_FEATURE_VERSIONmay be set to- when that
>=3.9is bumped to>=3.10, then that max-allowed version forNPY_FEATURE_VERSIONmoves up.
So would suggest dropping
| def_macros = [ | |
| # keep in sync with minimal runtime requirement (requirements.txt) | |
| ('NPY_TARGET_VERSION', 'NPY_1_19_API_VERSION') | |
| ] | |
| def_macros = [] |
| @@ -1 +1 @@ | |||
| numpy >= 1.13.3 | |||
| numpy >= 1.19.3 # keep in sync with NPY_TARGET_VERSION (setup.py) | |||
There was a problem hiding this comment.
Similarly the version constraint here could be dropped if we just want to support the oldest NumPy available on Python 3.9. Unless of course there are other reasons than NumPy C API to set a NumPy minimum
| numpy >= 1.19.3 # keep in sync with NPY_TARGET_VERSION (setup.py) | |
| numpy |
| /* Get the sizes of all the operands */ | ||
| dtypes_tmp = NpyIter_GetDescrArray(iter); | ||
| for (i = 0; i < n_inputs+1; ++i) { | ||
| self->memsizes[i] = dtypes_tmp[i]->elsize; | ||
| self->memsizes[i] = PyDataType_ELSIZE(dtypes_tmp[i]); | ||
| } |
There was a problem hiding this comment.
Wonder if it is possible to rewrite this with PyArray_ITEMSIZE as that works on NumPy 1 & 2 based on this release note
| @@ -1203,7 +1203,7 @@ NumExpr_run(NumExprObject *self, PyObject *args, PyObject *kwds) | |||
| Py_INCREF(dtypes[0]); | |||
| } else { // constant, like in '"foo"' | |||
| dtypes[0] = PyArray_DescrNewFromType(NPY_STRING); | |||
| dtypes[0]->elsize = (int)self->memsizes[1]; | |||
| PyDataType_SET_ELSIZE(dtypes[0], (npy_intp)self->memsizes[1]); | |||
There was a problem hiding this comment.
Wonder if there is a way to write this so it branches on NPY_ABI_VERSION. That way there could be NumPy 1 & 2 compilation paths
There was a problem hiding this comment.
The easiest way is to define the macro if it isn't defined. Without that users can't do a local build with build isolation on old NumPy (which won't allow to compile with numpy 1 and run in 2, that would be harder).
Clearer instructions: https://numpy.org/devdocs/numpy_2_0_migration_guide.html#the-pyarray-descr-struct-has-been-changed
I noticed while testing a downstream package that the following
ImportErrorwas raised from NumPy 2.0.0rc1 pointing to numexpr:Traceback
A module that was compiled using NumPy 1.x cannot be run in NumPy 2.1.0.dev0 as it may crash. To support both 1.x and 2.x versions of NumPy, modules must be compiled with NumPy 2.0. Some module may need to rebuild instead e.g. with 'pybind11>=2.12'. If you are a user of the module, the easiest solution will be to downgrade to 'numpy<2' or try to upgrade the affected module. We expect that some modules will need time to support NumPy 2. Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.12.2/x64/bin/pytest", line 8, in <module> sys.exit(console_main()) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 197, in console_main code = main() File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 174, in main ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main( File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall res = hook_impl.function(*args) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 332, in pytest_cmdline_main return wrap_session(config, _main) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 285, in wrap_session session.exitstatus = doit(config, session) or 0 File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 338, in _main config.hook.pytest_collection(session=session) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall res = hook_impl.function(*args) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 349, in pytest_collection session.perform_collect() File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 813, in perform_collect self.items.extend(self.genitems(node)) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 981, in genitems yield from self.genitems(subnode) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 981, in genitems yield from self.genitems(subnode) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 976, in genitems rep, duplicate = self._collect_one_node(node, handle_dupes) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 839, in _collect_one_node rep = collect_one_node(node) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/runner.py", line 565, in collect_one_node rep: CollectReport = ihook.pytest_make_collect_report(collector=collector) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__ return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall res = hook_impl.function(*args) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/runner.py", line 390, in pytest_make_collect_report call = CallInfo.from_call(collect, "collect") File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/runner.py", line 340, in from_call result: Optional[TResult] = func() File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/runner.py", line 388, in collect return list(collector.collect()) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/python.py", line 576, in collect self._register_setup_module_fixture() File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/python.py", line 589, in _register_setup_module_fixture self.obj, ("setUpModule", "setup_module") File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/python.py", line 315, in obj self._obj = obj = self._getobj() File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/python.py", line 573, in _getobj return importtestmodule(self.path, self.config) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/python.py", line 520, in importtestmodule mod = import_path( File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/pathlib.py", line 584, in import_path importlib.import_module(module_name) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/assertion/rewrite.py", line 178, in exec_module exec(co, module.__dict__) File "/home/runner/work/nonos/nonos/tests/test_plotting.py", line 4, in <module> import numexpr as ne File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/numexpr/__init__.py", line 24, in <module> from numexpr.interpreter import MAX_THREADS, use_vml, __BLOCK_SIZE1__ Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/numpy/core/_multiarray_umath.py", line 44, in __getattr__ raise ImportError(msg) ImportError: A module that was compiled using NumPy 1.x cannot be run in NumPy 2.1.0.dev0 as it may crash. To support both 1.x and 2.x versions of NumPy, modules must be compiled with NumPy 2.0. Some module may need to rebuild instead e.g. with 'pybind11>=2.12'. If you are a user of the module, the easiest solution will be to downgrade to 'numpy<2' or try to upgrade the affected module. We expect that some modules will need time to support NumPy 2.For more context:
oldest-supported-numpyis superseded by NumPy itself since version 1.25.0It is recommended by NumPy developers to plan releases of packages with Python extensions a some point between 2.0.0rc1 and 2.0.0 (final), which from the looks of it should be a couple months from now.