0.29.0: install target uses hardcoded install directory for python modules #4830

kloczek · 2023-12-07T16:24:57Z

from install target output:

running install
running build
running build_py
creating build
creating build/lib
creating build/lib/bcc
copying bcc/__init__.py -> build/lib/bcc
copying bcc/containers.py -> build/lib/bcc
copying bcc/disassembler.py -> build/lib/bcc
copying bcc/libbcc.py -> build/lib/bcc
copying bcc/perf.py -> build/lib/bcc
copying bcc/syscall.py -> build/lib/bcc
copying bcc/table.py -> build/lib/bcc
copying bcc/tcp.py -> build/lib/bcc
copying bcc/usdt.py -> build/lib/bcc
copying bcc/utils.py -> build/lib/bcc
copying bcc/version.py -> build/lib/bcc
running install_lib
creating /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib    <<<< FROM HERE
creating /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8
creating /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages
creating /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
copying build/lib/bcc/__init__.py -> /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
copying build/lib/bcc/containers.py -> /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
copying build/lib/bcc/disassembler.py -> /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
copying build/lib/bcc/libbcc.py -> /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
copying build/lib/bcc/perf.py -> /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
copying build/lib/bcc/syscall.py -> /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
copying build/lib/bcc/table.py -> /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
copying build/lib/bcc/tcp.py -> /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
copying build/lib/bcc/usdt.py -> /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
copying build/lib/bcc/utils.py -> /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
copying build/lib/bcc/version.py -> /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc
byte-compiling /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc/__init__.py to __init__.cpython-38.pyc
byte-compiling /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc/containers.py to containers.cpython-38.pyc
byte-compiling /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc/disassembler.py to disassembler.cpython-38.pyc
byte-compiling /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc/libbcc.py to libbcc.cpython-38.pyc
byte-compiling /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc/perf.py to perf.cpython-38.pyc
byte-compiling /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc/syscall.py to syscall.cpython-38.pyc
byte-compiling /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc/table.py to table.cpython-38.pyc
byte-compiling /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc/tcp.py to tcp.cpython-38.pyc
byte-compiling /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc/usdt.py to usdt.cpython-38.pyc
byte-compiling /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc/utils.py to utils.cpython-38.pyc
byte-compiling /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc/version.py to version.cpython-38.pyc
running install_egg_info
running egg_info
writing bcc.egg-info/PKG-INFO
writing dependency_links to bcc.egg-info/dependency_links.txt
writing top-level names to bcc.egg-info/top_level.txt
reading manifest file 'bcc.egg-info/SOURCES.txt'
writing manifest file 'bcc.egg-info/SOURCES.txt'
Copying bcc.egg-info to /home/tkloczko/rpmbuild/BUILDROOT/bcc-0.29.0-2.fc35.x86_64/lib/python3.8/site-packages/bcc-0.29.0-py3.8.egg-info
running install_scripts
writing list of installed files to '/home/tkloczko/rpmbuild/BUILD/bcc/x86_64-redhat-linux-gnu/install_manifest_python_bcc.txt'
/usr/lib/python3.8/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()

Instead hardcoding base directory it should be used output of

[tkloczko@pers-jacek SPECS]$ python3 -Ic "import sysconfig; print(sysconfig.get_path('purelib'))"
/usr/lib/python3.8/site-packages

or

[tkloczko@pers-jacek SPECS]$ python3 -Ic "import sysconfig; print(sysconfig.get_path('data'))"
/usr

The text was updated successfully, but these errors were encountered:

yonghong-song · 2023-12-07T18:33:41Z

Sorry. the tag is incorrect, it should be v0.29.0. could you sync with latest bcc repo and try again?

hhoffstaette · 2023-12-07T19:02:10Z

I have exactly the same problem on Gentoo using v0.29.0; python stuff is installed into /lib/python3.x/ (where it does not belong) and is not found by any tools.

yonghong-song · 2023-12-07T19:27:40Z

Could you help bisect which commit caused this issue?

hhoffstaette · 2023-12-07T20:30:49Z

The problem is caused by #6813fbce ("Allow for installing python as a non-system package").
Building 0.29.0-final with only that commit reverted installs python stuff where it should go, just like with 0.28.

kloczek · 2023-12-07T21:54:58Z

Could you help bisect which commit caused this issue?

OK. One sec ..

yonghong-song · 2023-12-07T22:57:17Z

@hhoffstaette @kloczek Could you help investigate how we could fix the issue at the same time preserving the functionality as in 6813fbc If needed we can add a cmake option for that. thanks!

kloczek · 2023-12-07T23:31:54Z

Issue is that 0.29.0 tag is without dist tar ball asset so I cannot plug that tag into my existing build procedure which I can request to execute on my build infra.

yonghong-song · 2023-12-08T00:00:05Z

@kloczek https://github.com/iovisor/bcc/releases/tag/v0.29.0 there is a tarball with submodules included.

eli-schwartz · 2023-12-08T02:30:29Z

@hhoffstaette @kloczek Could you help investigate how we could fix the issue at the same time preserving the functionality as in 6813fbc If needed we can add a cmake option for that. thanks!

I think that ideally you would stop calling one build system from another build system, but instead use one build system that knows how to handle both C/C++ and python.

Using setuptools isn't even a long-term solution either way, because ${PY_CMD} setup.py install ... is going to stop working when setuptools finally does what it's been warning it will do for a while now, and removes the ability to execute setup.py as a script. I'm not aware of a specific timeframe for this but it is something that they want to do at some point.

EDIT: for context, you can install a whole bunch more packages as well and then use those to install a setuptools-based wheel. However, this is a great deal of effort and additional dependencies for, ultimately, at the end of the day -- nothing other than "install the bcc/ directory to the correct site-packages".

Even the setuptools-produced .egg-info / .dist-info metadata is effectively useless, since it's not the same as PyPI.

hhoffstaette · 2023-12-08T05:08:30Z

@hhoffstaette @kloczek Could you help investigate how we could fix the issue at the same time preserving the functionality as in 6813fbc If needed we can add a cmake option for that. thanks!

The puzzling thing is that the existing construct if(NOT PYTHON_PREFIX) should work, but somehow does not. So I took a closer look at the conditional assignment and saw the comma between the variables, which looked fishy to me. I removed it (see below) and now it seems to work as expected. 🎉
Can someone more versed in CMake confirm that this won't break anything else?
Maybe @leeavital can chime in. If this is indeed correct I'll make a PR.

diff --git a/src/python/CMakeLists.txt b/src/python/CMakeLists.txt
index a897aa98..7f5b103e 100644
--- a/src/python/CMakeLists.txt
+++ b/src/python/CMakeLists.txt
@@ -38,7 +38,7 @@ foreach(PY_CMD ${PYTHON_CMD})
   add_custom_target(bcc_py_${PY_CMD_ESCAPED} ALL DEPENDS ${PIP_INSTALLABLE})
 
   if(NOT PYTHON_PREFIX)
-     set(PYTHON_PREFIX, ${CMAKE_INSTALL_PREFIX} )
+     set(PYTHON_PREFIX ${CMAKE_INSTALL_PREFIX})
   endif()
 
   install(

eli-schwartz · 2023-12-08T05:27:51Z

Can someone more versed in CMake confirm that this won't break anything else?

Using commas is incorrect CMake syntax. The Meson build system would require it, on the other hand. :P

Interestingly, you can see what cmake is actually doing here if you set it as a CACHE variable. The variable name is PYTHON_DIR,, but you cannot actually access such a variable after setting it, at least... not with ${} lookup syntax. It still works via some other methods not used here.

Unfortunately, cmake as a stringly-typed programming language has no innate protection against this kind of issue.

The spurious comma prevents the expression from working as intended. Fixes: iovisor@6813fbc Fixes: iovisor#4830 Signed-off-by: Holger Hoffstätte <[email protected]>

The spurious comma prevents the expression from working as intended. Fixes: 6813fbc Fixes: #4830 Signed-off-by: Holger Hoffstätte <[email protected]>

yonghong-song · 2023-12-08T06:55:05Z

@hhoffstaette @eli-schwartz @kloczek @rtomayko Thanks for all of you trying to fix the problem! Just merged to the master branch. Do you think we need another release (e.g., v0.29.1) containing the fix? Or we can wait for another say two months for the next release (e.g., v0.30.0)?

eli-schwartz · 2023-12-08T06:57:27Z

I would recommend making another release since build issues that prevent people from successfully installing the software tend to be a pretty thorny problem for affected users.

Two months is a bit on the long side, for that.

yonghong-song · 2023-12-08T06:58:35Z

Sounds good. Will cut another minor release tomorrow.

Bug: iovisor/bcc#4823 Bug: iovisor/bcc#4830 Reported-by: Holger Hoffstätte <[email protected]> Signed-off-by: Sam James <[email protected]>

hhoffstaette · 2023-12-08T07:31:45Z

Sounds good. Will cut another minor release tomorrow.

Appreciate it, but please also add the python-3.12 fix in #4832 - thanks :)

hhoffstaette mentioned this issue Dec 8, 2023

python/cmake: remove extraneous comma between variables #4833

Merged

yonghong-song closed this as completed in #4833 Dec 8, 2023

yonghong-song pushed a commit that referenced this issue Dec 8, 2023

python/cmake: remove extraneous comma between variables

4490b7e

The spurious comma prevents the expression from working as intended. Fixes: 6813fbc Fixes: #4830 Signed-off-by: Holger Hoffstätte <[email protected]>

gentoo-bot pushed a commit to gentoo/gentoo that referenced this issue Dec 8, 2023

profiles: mask broken =dev-util/bcc-0.29.0

91f1ac0

Bug: iovisor/bcc#4823 Bug: iovisor/bcc#4830 Reported-by: Holger Hoffstätte <[email protected]> Signed-off-by: Sam James <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

0.29.0: install target uses hardcoded install directory for python modules #4830

0.29.0: install target uses hardcoded install directory for python modules #4830

kloczek commented Dec 7, 2023

yonghong-song commented Dec 7, 2023

hhoffstaette commented Dec 7, 2023

yonghong-song commented Dec 7, 2023

hhoffstaette commented Dec 7, 2023

kloczek commented Dec 7, 2023

yonghong-song commented Dec 7, 2023

kloczek commented Dec 7, 2023

yonghong-song commented Dec 8, 2023

eli-schwartz commented Dec 8, 2023 •

edited

Loading

hhoffstaette commented Dec 8, 2023

eli-schwartz commented Dec 8, 2023

yonghong-song commented Dec 8, 2023

eli-schwartz commented Dec 8, 2023 •

edited

Loading

yonghong-song commented Dec 8, 2023

hhoffstaette commented Dec 8, 2023

0.29.0: install target uses hardcoded install directory for python modules #4830

0.29.0: install target uses hardcoded install directory for python modules #4830

Comments

kloczek commented Dec 7, 2023

yonghong-song commented Dec 7, 2023

hhoffstaette commented Dec 7, 2023

yonghong-song commented Dec 7, 2023

hhoffstaette commented Dec 7, 2023

kloczek commented Dec 7, 2023

yonghong-song commented Dec 7, 2023

kloczek commented Dec 7, 2023

yonghong-song commented Dec 8, 2023

eli-schwartz commented Dec 8, 2023 • edited Loading

hhoffstaette commented Dec 8, 2023

eli-schwartz commented Dec 8, 2023

yonghong-song commented Dec 8, 2023

eli-schwartz commented Dec 8, 2023 • edited Loading

yonghong-song commented Dec 8, 2023

hhoffstaette commented Dec 8, 2023

eli-schwartz commented Dec 8, 2023 •

edited

Loading

eli-schwartz commented Dec 8, 2023 •

edited

Loading