Safer handling of internal soname dependencies

[pmatilai]

This is one of those topics that gets raised semi-annually at least. To point out a couple, https://bugzilla.redhat.com/show_bug.cgi?id=2259260 and https://bugzilla.redhat.com/show_bug.cgi?id=1464368 and #1227 but there are certainly more out there.

For some cases like an unfortunately named project starting with lib having a Python/Perl module, the provide may be mostly harmless if ugly. The far worse issue is bundled libraries that are incompatible with the system ones and not even in the path, but create identical provide names which a depsolver may end up pulling instead of the system level library, and break things quite badly.

The fundamental issue may well be unsolvable, the main focus here is to look for a solution that makes those bundled libraries less dangerous. If other problems get solved while we're at it, great.

The common suggestion is to filter out soname dependencies outside the system library path, but this has multiple problems: filtering them on the rpm metadata level doesn't make the dependencies go away, and they may well be needed for sub-packages and so on. What may work better is transforming the sonames dependencies for libraries outside system path to something else, perhaps including the parent package name in the name so it can't be mixed with the system or somebody elses bundled library. For provides one can do this by parsing /etc/ld.so.conf[.d] but requires seems to require solving, which is something elfdeps doesn't (and IMO, shouldn't) do. (I don't like parsing other people's config files either, but that seems unavoidable)

One simple possibility would be giving elfdeps a list/regex of soname prefixes to match. Eg, passing "libxml2" would cause it to map any dependencies found on libxml* to some other, internal-suggesting name. Such as %{name}-libxml*. That list/regex could be set by the packager, but to satisfy the goal of bundled library safety, it needs to happen automatically for provides. I guess something like this would do:

1. figure out system library path, considering ld.so.conf[.d] in the buildroot and everything
2. find out any shared libraries in the buildroot outside that path
3. set the elfdeps mangler (eg %{name}-) to apply to everything found in 2
4. profit?

Mangling discovered dependencies isn't hard, retrofitting the discovery parts to the rpm framerwork probably is.

[codonell]

@pmatilai Can you expand a bit on what you mean by not liking parsing other people's config files? Is it that you would like a clean way to discover the system library paths searched by the dynamic loader?

[codonell]

@pmatilai Just for clarity we are leaving out DT_RPATH, environmental combination of LD_LIBRARY_PATH and the search paths (packages that alter shell defaults and global search paths), LD_PRELOAD interposition (similar to LD_LIBRARY_PATH but early loads the SONAME), and DT_RUNPATH and their effect on the resolution of the absolute path used to load the DSO requried by either an executable or library. These are AFAIK details that RPM's elfdeps does not model and so still requires the developer to express them directly as manual requires or provides. This is not a regression in behaviour just a modelling choice. After all our intent is to make this easier for developers without needing to expose all of the details around the ELF standard and the current Linux implementations.

Yes, I agree that the host (or mock chroot) ld.so.conf[.d] paths should be searched for requires. The still broken case is when the rpm installs a DSO into a path created by an rpm in the Requires: (rather than BuildRequire:) since this cannot be discovered at build time? In this case the rpm neither provides the ld.so.conf fragment, nor is the DSO used at build time. Is this a narrow enough case that we can ignore it? Initially my answer is: Yes, I think we should focus solely on the host ld.so.conf[.d] paths and the paths provided by the package itself (as it is being built)

It seems natural to use ldconfig to provide these paths? We could add a new option to just print directories searched.

[pmatilai]

Yeah elfdeps basically ignores all path-aspects, it only really handles DT_NEEDED and DT_SONAME.
One thing here though: a manually added require/provide is not equal to one discovered by rpm, the latter are tracked on file-level and rpm actually uses this information. That's why I'm so reluctant to filter out anything and rather turn them into something non-harmful instead (eg prefix discovered provides/requires by package name or such). And having packagers form those wacko ()(64bit) markers manually doesn't really fly either.

The paths from the config is actually what I've been after all along, I hadn't really even considered there were other paths at play (silly me, of course the are). But yeah having an ldconfig option to print out the configured paths would be nice. If we can get it to print out the host config directories, we can use that to create a temporary config file to use inside the buildroot to include any fragments from there and print again to get the final list of paths for a package.

As for the Requires vs Buildrequires matter, I'd expect the vast majority of these cases to be covered by the foo-devel buildrequire to pull in the part with the ld.so.conf too. The corner-case where that doesn't happen (it's possible of course) can be worked around by just adding a Buildrequire on the thing with the ld.so.conf.d fragment, easy enough.

[pmatilai]

Actually, we'll need to have this path-discovery wrapped in some libc specific helper script in rpm because this eg musl has an entirely different library search path mechnism. That's fine though, for glibc we could implement our own ld.so.conf parser for the immediate use and then once the ldconfig native version is widely available, switch to that for glibc.

[pmatilai]

One thing just for the record, up to now elfdeps has been path agnostic by design to match the dynamic linker: it doesn't matter where those libraries come from as long as they're in the runtime path. Of course it's not an "end user tool" but having consistent output regardless of the file location is nice really. So I need to think/investingate whether the path dependent parts wouldn't be better done some place else, such as a separate filter/transform post step.

[codonell]

@pmatilai I agree that while a "search path" is a fundamental part of the model for the full view of the dynamic loader, it need not be part of rpm's elfdeps and could become some kind of filter option that is driven by libc specific search information.

[praiskup]

Seems related https://bugzilla.redhat.com/show_bug.cgi?id=2266553

[codonell]

@praiskup It is exactly the same problem. We will continue to have these issues at the distribution level until we change the way we model how a package exports an interface e.g. list of shared objects. There is perhaps a broader question that this mistake would not have been caught without the current semantics we have, but the flip side is that it fails so badly that it blocks work being done by others and is disruptive. While the failure would have been less spectacular we could have caught this in CI gating by detecting that this package was doing something wrong.

[praiskup]

Agreed, especially with the CI gating part.