BUG: f2py cannot find .mod files automatically for the new meson backend #27018

1234zou · 2024-07-23T17:26:06Z

Describe the issue:

I'm trying to compile my Fortran code using f2py. For python>=3.12, the meson backend is now the default. But meson/f2py cannot automatically find the generated .mod files in the current directory. These .mod files could be automatically found for older versions of python and distutils backend. By the way, gcc/gfortran in this machine is 4.8.5.

Thanks for any kind help and suggestion!

Best,
Jingxiang

Reproduce the code example:

A simple example is shown using two Fortran files and one Python script:

here is a.f90

module para
 implicit none
 integer, parameter :: n = 5
end module para

b.f90

subroutine prt_n
 use para, only: n
 implicit none
 write(6,'(A,I2)') 'n=', n
end subroutine prt_n

Compiling commands are

gfortran a.f90 -c
f2py -c a.o b.f90 -m test

The first command would generate a.o and para.mod files in the current directory. para.mod can be automatically detected for the distutils backend, but not for the meson backend.

test.py

from test import prt_n
prt_n()

Assuming that the test.xxx.so is generated (which not True for the meson backend), the result would be n= 5.

Error message:

After running f2py -c a.o b.f90 -m test, the error message is

Found ninja-1.10.2 at /public/home/jxzou/software/anaconda3/envs/py312/bin/ninja
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /public/home/jxzou/software/anaconda3/envs/py312/bin/ninja -C /public/home/jxzou/tmp/tmpgljbyueg/bbdir
ninja: Entering directory `/public/home/jxzou/tmp/tmpgljbyueg/bbdir'
[3/6] Compiling Fortran object test.cpython-312-x86_64-linux-gnu.so.p/b.f90.o
FAILED: test.cpython-312-x86_64-linux-gnu.so.p/b.f90.o
gfortran -Itest.cpython-312-x86_64-linux-gnu.so.p -I. -I.. -I/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/core/include -I/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/src -I/public/home/jxzou/software/anaconda3/envs/py312/include/python3.12 -fvisibility=hidden -D_FILE_OFFSET_BITS=64 -Wall -O3 -fPIC -Jtest.cpython-312-x86_64-linux-gnu.so.p -o test.cpython-312-x86_64-linux-gnu.so.p/b.f90.o -c ../b.f90
../b.f90:2.5:

 use para, only: n
     1
Fatal Error: Can't open module file 'para.mod' for reading at (1): No such file or directory
[5/6] Compiling C object test.cpython-312-x86_64-linux-gnu.so.p/3fa16a5cf62c3f6199e0fa73eb3f172c17c452f1_.._.._f2py_src_fortranobject.c.o
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/public/home/jxzou/software/anaconda3/envs/py312/bin/f2py", line 11, in <module>
    sys.exit(main())
             ^^^^^^
  File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 766, in main
    run_compile()
  File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 738, in run_compile
    builder.compile()
  File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 178, in compile
    self.run_meson(self.build_dir)
  File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 173, in run_meson
    self._run_subprocess_command(compile_command, build_dir)
  File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 167, in _run_subprocess_command
    subprocess.run(command, cwd=cwd, check=True)
  File "/public/home/jxzou/software/anaconda3/envs/py312/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['meson', 'compile', '-C', 'bbdir']' returned non-zero exit status 1.

Python and NumPy Versions:

python>=3.12 and numpy>=1.26

Runtime Environment:

[{'numpy_version': '1.26.4',
  'python': '3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, '
            '15:12:24) [GCC 11.2.0]',
  'uname': uname_result(system='Linux', node='mu003', release='3.10.0-862.el7.x86_64', version='#1 SMP Fri Apr 20 16:44:24 UTC 2018', machine='x86_64')},
 {'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
                      'found': ['SSSE3',
                                'SSE41',
                                'POPCNT',
                                'SSE42',
                                'AVX',
                                'F16C',
                                'FMA3',
                                'AVX2',
                                'AVX512F',
                                'AVX512CD',
                                'AVX512_SKX',
                                'AVX512_CLX',
                                'AVX512_CNL',
                                'AVX512_ICL'],
                      'not_found': ['AVX512_KNL', 'AVX512_KNM']}},
 {'filepath': '/public/home/jxzou/software/anaconda3/envs/py312/lib/libmkl_rt.so.2',
  'internal_api': 'mkl',
  'num_threads': 64,
  'prefix': 'libmkl_rt',
  'threading_layer': 'intel',
  'user_api': 'blas',
  'version': '2023.1-Product'},
 {'filepath': '/public/home/jxzou/software/anaconda3/envs/py312/lib/libiomp5.so',
  'internal_api': 'openmp',
  'num_threads': 64,
  'prefix': 'libiomp',
  'user_api': 'openmp',
  'version': None}]

Context for the issue:

I'm a developer of the open source package MOKIT. By using f2py, we offer lots of convenient and efficient Python APIs to users in the computational chemistry field. We are grateful for your numpy f2py functionalities. MOKIT source code are mainly written using Fortran, so we heavily reply on the f2py to obtain corresponding .so files.

Currently my workaround is to change all .o filenames into .f90 ones in f2py commands, e.g.

f2py -c a.f90 b.f90 -m test

This is a tiny change for this simple example. But a somewhat big work to do in MOKIT. And this change would make test.so contain more APIs than previous. For example, if there are many subroutines in a.f90, it will be all embraced into test.so. This problem does not exist when the distutils backend is used previously. I do not know whether this is a feature or bug for the new meson backend. I'm wondering if there is any elegant solution.

The text was updated successfully, but these errors were encountered:

1234zou · 2024-07-24T12:34:37Z

Hi, @HaoZeke , would you please add the tag component: numpy.f2py for this issue? It seems that I cannot add this tag by myself. Thanks a lot.

HaoZeke · 2024-07-24T12:44:21Z

Thanks for the issue and ping.

I'm thinking that there is probably no very good default solution here, since the meson backend doesn't have any information on the other .f90 or .mod files unless they are part of the f2py call right now (which is not desirable due to API scope creep as noted). The best way would be to use --builddir and modify the generated meson.build files to compile the Fortran codes (like param) and link them to the .so extension.

To make this automatic, something which we could support but really probably shouldn't is to take a list of fortran files to be compiled and linked but not otherwise processed by F2PY. In general though, for all but trivial cases this will not be enough so it doesn't make a lot of sense as a feature.

I think, with the new backend projects are slightly encouraged to transition the entire build system to meson for ease of use, but the documentation definitely should cover all this.

Another option is to try to pass the .mod files directly as object files, or as link arguments, will look into that.

1234zou · 2024-07-24T13:58:59Z

Oh, thanks for the quick and detailed response! My personal viewpoint is: if meson backend cannot auto-detect .mod files, it would be better to provide a command line argument, say -moddir, which tells meson the location of .mod files, in the future f2py. Just a suggestion.

Thanks again for your kind help.

P-Kaempf · 2024-08-09T08:15:48Z

The issue here is that the module cannot be found if there is no executable code inside. I had the same problem and posted a question to Stackexchange: https://stackoverflow.com/questions/78839287/f2py-in-numpy-2-0-1-does-not-expose-variables-the-way-numpy-1-26-did-how-can-i/78851726#78851726

An ugly workaround is to simply add some executable code inside the module, so it contains not only assignments. This got me unstuck. Just follow the answer on Stackexchange.

And I still think this is a bug, but it can be worked around.

HaoZeke · 2024-08-09T09:56:52Z

The issue here is that the module cannot be found if there is no executable code inside. I had the same problem and posted a question to Stackexchange: https://stackoverflow.com/questions/78839287/f2py-in-numpy-2-0-1-does-not-expose-variables-the-way-numpy-1-26-did-how-can-i/78851726#78851726

An ugly workaround is to simply add some executable code inside the module, so it contains not only assignments. This got me unstuck. Just follow the answer on Stackexchange.

And I still think this is a bug, but it can be worked around.

No that's not really the issue discussed here. Adding executable code makes F2PY bind it, which has the same problem as discussed in the original code, i.e. there will be bindings which shouldn't be exposed to the user.

Arguably the right fix is not the stackexchange answer (might answer there later) but as discussed to make the project skeleton, and modify the meson.build to build the fortran module, before linking it to the target generated by f2py.

EDIT: The stackexchange question is actually separate and would be a bug, if the user passes a variable only fortran file, F2PY should wrap it, could you open another issue for that @P-Kaempf?

P-Kaempf · 2024-08-10T00:32:46Z

The Stackexchange answer is for a specific question. Maybe it is not exactly your question, but the answer still holds true for the original question. Did you really downvote it? That helps nobody.

HaoZeke · 2024-08-10T12:07:20Z

The Stackexchange answer is for a specific question. Maybe it is not exactly your question, but the answer still holds true for the original question. Did you really downvote it? That helps nobody.

That was hasty, and reverted (though it is worth noting that it was irrelevant to this issue). Please open another issue related to the question.

pescap · 2024-09-22T15:31:25Z

Dear all,
I am facing the same issue:

gfortran a.f90 -c
f2py -c a.o b.f90 -m test

Unfortunately, my a.f90 file includes custom data structures that cannot be compiled via f2py (e.g., type(curve)), but that I need to import to run f2py b.f90.

Everything was working fine with distutils, but I have not found a solution with meson. Do you suggest that I wait before migrating?

Thank you for your help,

HaoZeke · 2024-09-23T18:23:13Z

Dear all, I am facing the same issue:
gfortran a.f90 -c
f2py -c a.o b.f90 -m test
Unfortunately, my a.f90 file includes custom data structures that cannot be compiled via f2py (e.g., type(curve)), but that I need to import to run f2py b.f90.

Everything was working fine with distutils, but I have not found a solution with meson. Do you suggest that I wait before migrating?

Thank you for your help,

Could you please provide a minimum working example showcasing the bug so it can be reproduced? In theory if the type(curve) part is not part of the F2PY call it should be fine, e.g. see the use of types in https://github.com/HaoZeke/GaussJacobiQuad

1234zou added the 00 - Bug label Jul 23, 2024

HaoZeke added the component: numpy.f2py label Jul 24, 2024

HaoZeke changed the title ~~f2py cannot find .mod files automatically for the new meson backend~~ BUG: f2py cannot find .mod files automatically for the new meson backend Jul 24, 2024

HaoZeke added the 04 - Documentation label Jul 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: f2py cannot find .mod files automatically for the new meson backend #27018

BUG: f2py cannot find .mod files automatically for the new meson backend #27018

1234zou commented Jul 23, 2024 •

edited

Loading

1234zou commented Jul 24, 2024

HaoZeke commented Jul 24, 2024 •

edited

Loading

1234zou commented Jul 24, 2024 •

edited

Loading

P-Kaempf commented Aug 9, 2024

HaoZeke commented Aug 9, 2024 •

edited

Loading

P-Kaempf commented Aug 10, 2024 •

edited

Loading

HaoZeke commented Aug 10, 2024

pescap commented Sep 22, 2024 •

edited

Loading

HaoZeke commented Sep 23, 2024

BUG: f2py cannot find .mod files automatically for the new meson backend #27018

BUG: f2py cannot find .mod files automatically for the new meson backend #27018

Comments

1234zou commented Jul 23, 2024 • edited Loading

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

1234zou commented Jul 24, 2024

HaoZeke commented Jul 24, 2024 • edited Loading

1234zou commented Jul 24, 2024 • edited Loading

P-Kaempf commented Aug 9, 2024

HaoZeke commented Aug 9, 2024 • edited Loading

P-Kaempf commented Aug 10, 2024 • edited Loading

HaoZeke commented Aug 10, 2024

pescap commented Sep 22, 2024 • edited Loading

HaoZeke commented Sep 23, 2024

1234zou commented Jul 23, 2024 •

edited

Loading

HaoZeke commented Jul 24, 2024 •

edited

Loading

1234zou commented Jul 24, 2024 •

edited

Loading

HaoZeke commented Aug 9, 2024 •

edited

Loading

P-Kaempf commented Aug 10, 2024 •

edited

Loading

pescap commented Sep 22, 2024 •

edited

Loading