Error while loading shared libraries: libgdal.so.32 for Python 3.10/Numpy 1.23 tests

[weiji14]

Description of the problem

Scheduled unit tests are failing for Linux, macOS and Windows for Python 3.10/NumPy 1.23, and also the GMT Dev Tests for Windows.

This seems to be due to GDAL version is no longer pinned after conda-forge/gmt-feedstock#226, causing GDAL 3.5 to be accidentally installed in the environment, though GMT is built with GDAL 3.6. Compare the previously working linux-64/gmt-6.4.0-h78e8b95_2.tar.bz2 (which has gdal >=3.5.1,<3.6.0a0) with the newer linux-64/gmt-6.4.0-hb5a752b_3.conda (which doesn't pin to a specific gdal version).

Minimal Complete Verifiable Example

See sample traceback at https://github.com/GenericMappingTools/pygmt/actions/runs/3560839755/jobs/5981192688#step:11:30

Full error message

gmt: error while loading shared libraries: libgdal.so.32: cannot open shared object file: No such file or directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/runner/work/pygmt/pygmt/pygmt/__init__.py", line 74, in <module>
    _begin()
  File "/home/runner/work/pygmt/pygmt/pygmt/session_management.py", line 16, in begin
    with Session() as lib:
  File "/home/runner/work/pygmt/pygmt/pygmt/clib/session.py", line 191, in __enter__
    self.create("pygmt-session")
  File "/home/runner/work/pygmt/pygmt/pygmt/clib/session.py", line 338, in create
    c_create_session = self.get_libgmt_func(
  File "/home/runner/work/pygmt/pygmt/pygmt/clib/session.py", line 290, in get_libgmt_func
    self._libgmt = load_libgmt()
  File "/home/runner/work/pygmt/pygmt/pygmt/clib/loading.py", line 59, in load_libgmt
    raise GMTCLibNotFoundError("\n".join(error_msg))
pygmt.exceptions.GMTCLibNotFoundError: Error loading GMT shared library at 'libgmt.so'.
libgdal.so.32: cannot open shared object file: No such file or directory

System information

OS: ubuntu-22.04
PYTHON: 3.10
NUMPY: 1.23

[weiji14]

Hmm, it seems like gmt-6.4.0-h8b1eae8_5.conda is still being favoured over gmt-6.4.0-h8b1eae8_6.conda (which contains the GDAL pinning fix done in conda-forge/gmt-feedstock#232), e.g. at https://github.com/GenericMappingTools/pygmt/actions/runs/3570223001/jobs/6000995039#step:5:148. I'm not sure why the mamba solver isn't picking up the newer build number 6, is there some dependency that was pinned too tightly?

[seisman]

Just tried to create a fresh environment on macOS:

The following command install GMT 6.4.0 build 6 and GDAL 3.6.0:

mamba create --name pygmt python=3.10 numpy pandas xarray netcdf4 packaging gmt

but installing geopandas shows GMT 6.4.0 build 5 and GDAL 3.5.3:

mamba create --name pygmt python=3.10 numpy pandas xarray netcdf4 packaging gmt geopandas

So, maybe geopandas for Python 3.10 is not compiled with GDAL 3.6?

Also tried the following command:

mamba create --name pygmt python=3.10 numpy pandas xarray netcdf4 packaging gmt geopandas 'gdal>=3.6.0'
Encountered problems while solving:
  - package gdal-3.6.0-py310he22d2f2_0 requires libgdal 3.6.0 h1951153_0, but none of the providers can be installed

Does this mean the GDAl itself is problematic?

[seisman]

More tests:

• mamba create --name pygmt python=3.10 geopandas installs gdal 3.5.3
• mamba create --name pygmt python=3.10 gdal geopandas installs gdal 3.5.3

$ mamba create --name pygmt python=3.10 geopandas 'gdal>=3.6.0'
Encountered problems while solving:
  - nothing provides gdal 1.11.4 needed by fiona-1.6.3-np110py27_0

Perhaps we have to wait for conda-forge/fiona-feedstock#202.

[weiji14]

A little tempted to submit a repodata patch to https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/tree/main/recipe to update the gmt conda-forge package's metadata to specify the correct GDAL pinning. Would need to patch GMT 6.4.0 build 3, 4 and 5 to use gdal >=3.6,<3.7 as in build 6.

Otherwise, another workaround is to tell people to use GMT 6.4.0 build 2 (which is built with GDAL 3.5). E.g. by running an install command like:

• conda install -c conda-forge gmt=6.4.0=h78e8b95_2 pygmt for Linux
• conda install -c conda-forge gmt-6.4.0=hace14ca_2 pygmt for Windows
• conda-install -c conda-forge gmt=6.4.0=h216c163_2 pygmt for osx-arm64 (Apple silicon chips)
• conda install -c conda-forge gmt=6.4.0=h81e1e63_2 pygmt for osx (Intel chips)

[seisman]

A little tempted to submit a repodata patch to conda-forge/conda-forge-repodata-patches-feedstock@main/recipe to update the gmt conda-forge package's metadata to specify the correct GDAL pinning. Would need to patch GMT 6.4.0 build 3, 4 and 5 to use gdal >=3.6,<3.7 as in build 6.

Otherwise, another workaround is to tell people to use GMT 6.4.0 build 2 (which is built with GDAL 3.5). E.g. by running an install command like:

• conda install -c conda-forge gmt=6.4.0=h78e8b95_2 pygmt for Linux
• conda install -c conda-forge gmt-6.4.0=hace14ca_2 pygmt for Windows
• conda-install -c conda-forge gmt=6.4.0=h216c163_2 pygmt for osx-arm64 (Apple silicon chips)
• conda install -c conda-forge gmt=6.4.0=h81e1e63_2 pygmt for osx (Intel chips)

Sounds good.

[weiji14]

A little tempted to submit a repodata patch to conda-forge/conda-forge-repodata-patches-feedstock@main/recipe to update the gmt conda-forge package's metadata to specify the correct GDAL pinning. Would need to patch GMT 6.4.0 build 3, 4 and 5 to use gdal >=3.6,<3.7 as in build 6.

Sounds good.

Ok, patch submitted at conda-forge/conda-forge-repodata-patches-feedstock#366.

[weiji14]

Ok, patch submitted at conda-forge/conda-forge-repodata-patches-feedstock#366.

Repodata patch has been merged. I just tried running mamba create --name pygmt python=3.10 numpy pandas xarray netcdf4 packaging gmt geopandas and it is pulling down GMT 6.4.0 build 2 with GDAL 3.5 as expected. Closing this as resolved.