Trying to understand the relationship between LD_LIBRARY_PATH and conda/mamba environments

[mpizenberg]

I'm using a library, that has some looking for libpython3.10.so.1.0. This is a python frontend for a C library, so not sure if that plays a role here. The issue I have is that I only install python in dedicated environments which means I don't have anything in my LD_LIBRARY_PATH leading to such a shared library.

I can fix that issue by adding $HOME/miniforge3/envs/my_env/lib to LD_LIBRARY_PATH but now I'm wondering the following.

1. Is that library I'm depending on wrong to look for libpython3.10.so.1.0
2. If not, why doing something like mamba activate my_env does not add $HOME/miniforge3/my_env/lib to LD_LIBRARY_PATH?
3. Is there a cleaner / automatic way to do that? I don't want to manually add to LD_LIBRARY_PATH everytime I need it. And I don't want to add it to my config files such that it would be there even when I've not activated that environment.

[jaimergp]

• Where in the conda environment prefix is your library so located?
• What's the output of ldd <yourlibrary.so>?

[mpizenberg]

My library so is not within the conda environment as it's not installed by the wheel. I decompress the library inside /opt/ids-peak_2.3.0.0-15823_amd64. So there is a bin/ and a lib/ directory inside these dirs. I've add the bin dir to my PATH and added the lib dir to a config file inside /etc/ld.so.conf.d/ so that the lib is available:

❯ strings -n5 /etc/ld.so.cache | grep ids_peak
/opt/ids-peak_2.3.0.0-15823_amd64/lib/libids_peak.so
...

This is the output of ldd:

❯ ldd libids_peak.so
        linux-vdso.so.1 (0x00007ffc031b6000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007ff58682c000)
        libFirmwareUpdate_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/./libFirmwareUpdate_gcc63_v3_2_IDS.so (0x00007ff586000000)
        libGCBase_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/./libGCBase_gcc63_v3_2_IDS.so (0x00007ff585c00000)
        libGenApi_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/./libGenApi_gcc63_v3_2_IDS.so (0x00007ff585600000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007ff586827000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007ff585200000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007ff586738000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007ff5863db000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007ff585e1e000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007ff586850000)
        libLog_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/./libLog_gcc63_v3_2_IDS.so (0x00007ff584e00000)
        libMathParser_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/./libMathParser_gcc63_v3_2_IDS.so (0x00007ff584a00000)
        libXmlParser_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/./libXmlParser_gcc63_v3_2_IDS.so (0x00007ff584600000)
        libNodeMapData_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/./libNodeMapData_gcc63_v3_2_IDS.so (0x00007ff584200000)

Then after creating and activating the ids_peak mamba environment, I pip install the python wheel for the lib.

ids-peak-python on  main [?] is 📦 v0.1.1 via 🐍 v3.10.13 via 🅒 ids_peak
❯ pip install /opt/ids-peak_2.3.0.0-15823_amd64/local/share/ids/bindings/python/wheel
/ids_peak-1.5.0.0-cp310-cp310-linux_x86_64.whl
Processing /opt/ids-peak_2.3.0.0-15823_amd64/local/share/ids/bindings/python/wheel/ids_peak-1.5.0.0-cp310-cp310-linux_x86_64.whl
Installing collected packages: ids-peak
Successfully installed ids-peak-1.5.0.0

Then I try to import module (named the same) inside the ids_peak package:

ids-peak-python on  main [?] is 📦 v0.1.1 via 🐍 v3.10.13 via 🅒 ids_peak took 3m45s
❯ python
Python 3.10.13 | packaged by conda-forge | (main, Oct 26 2023, 18:07:37) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ids_peak
>>> from ids_peak import ids_peak
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/matthieu/miniforge3/envs/ids_peak/lib/python3.10/site-packages/ids_peak/ids_peak.py", line 18, in <module>
    from . import _ids_peak_python_interface
ImportError: libpython3.10.so.1.0: cannot open shared object file: No such file or directory
>>>

[mpizenberg]

But if I add to LD_LIBRARY_PATH the lib/ directory of the mamba environment, then all is working fine:

❯ echo $LD_LIBRARY_PATH
/home/matthieu/miniforge3/envs/ids_peak/lib

❯ python
Python 3.10.13 | packaged by conda-forge | (main, Oct 26 2023, 18:07:37) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ids_peak import ids_peak
>>>

[mpizenberg]

When I look at the installed wheel directory inside the environment (/home/matthieu/miniforge3/envs/ids_peak/lib/python3.10/site-packages/ids_peak/) I can see these two files:

ids_peak.py
_ids_peak_python_interface.so

And when I call ldd on that so file, I get:

❯ ldd _ids_peak_python_interface.so
        linux-vdso.so.1 (0x00007fff84960000)
        libids_peak.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/libids_peak.so (0x00007f015b600000)
        libpython3.10.so.1.0 => not found
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f015b200000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f015c078000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f015c053000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f015b01e000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f015c184000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f015c04c000)
        libFirmwareUpdate_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/libFirmwareUpdate_gcc63_v3_2_IDS.so (0x00007f015ac00000)
        libGCBase_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/libGCBase_gcc63_v3_2_IDS.so (0x00007f015a800000)
        libGenApi_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/libGenApi_gcc63_v3_2_IDS.so (0x00007f015a200000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f015c045000)
        libLog_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/libLog_gcc63_v3_2_IDS.so (0x00007f0159e00000)
        libMathParser_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/libMathParser_gcc63_v3_2_IDS.so (0x00007f0159a00000)
        libXmlParser_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/libXmlParser_gcc63_v3_2_IDS.so (0x00007f0159600000)
        libNodeMapData_gcc63_v3_2_IDS.so => /opt/ids-peak_2.3.0.0-15823_amd64/lib/libNodeMapData_gcc63_v3_2_IDS.so (0x00007f0159200000)

Notice the libpython3.10.so.1.0 => not found.

[isuruf]

You shouldn't link to libpython3.so in extensions. You should link only when you are embedding python in another application. (eg: C/C++)
See mesonbuild/meson#7712 (comment)

[mpizenberg]

@isuruf So what you say is the camera maker (IDS) should not link their _ids_peak_python_interface.so library against libpython3.so. So it's mis-usage on their part.

Do you know of a workaround on my part? Are there hooks somewhere in the environment configuration where I could automatically add the libpython parent dir to my LD_LIBRARY_PATH and then clear it after deactivating the env?