Require python3- prefix with PyPI keys

[sloretz]

Discussion

#25474 (comment)
#23836 (comment)

I think it would make sense to avoid python- or python3- prefixes on PyPI only keys because it could give a mistaken impression that it determines the Python version used.
The prefix doesn't make a difference because the actual python version depends on ROS_PYTHON_VERSION which itself is set by the ROS distro.

https://www.ros.org/reps/rep-0151.html#rosdep-and-pip-packages.

I think it would be confusing if a package in ROS Melodic workspace depended on python3-foobar-pip because rosdep would install it for Python 2.

[nuclearsandwich]

This makes sense for PyPI only keys since migration from a -pip key to a system package key will require changing the key no matter what.

Does it make sense to explicitly state that when PyPI packages are being used to "fill in" on systems where a distribution default package is unavailable the python3?- key format should still be used.

[tfoote]

The python2 vs python3 ambiguity is suboptimal. However one of the other major reasons for keeping the python prefix is to keep the sorting the same. We really don't want to have pip rules overriding the apt rules. And if you have python3-foo it's much harder to double check that foo-pip doesn't exist some hundreds of lines away in the database. Whereas if you have python3-foo and python3-foo-pip they are side by side and are trivial to identify as conflicting.

Another significant advantage of having the names the same is that when/if things get added to an apt repository in the future the new rosdep key is just removing the '-pip' extension which is easier for maintainers and clear what the substitute should be.

This also is following our standard to use the debian package naming conventions which is valuable for consistency even if the package isn't on debian platforms it's consistent and people can rely on it compared to supporting all the pip variations. We do the same for gentoo and arch who have been dealing with this ambiguity for much longer being python3 by default for quite a while without much problem.

Looking forward Melodic is the last python2 specific rosdistro we have so that mismatch issue will go away. And we will just use python3 for everything. But the conflicts between pip and apt will not so I'd strongly recommend sticking to using the prefixes and possibly saying that everything pip explicit should be added using the python3 prefix to not have duplicate rules as it will be eclipsed shortly.

[sloretz]

. We really don't want to have pip rules overriding the apt rules. And if you have python3-foo it's much harder to double check that foo-pip doesn't exist some hundreds of lines away in the database. Whereas if you have python3-foo and python3-foo-pip they are side by side and are trivial to identify as conflicting.

I don't think I understand this part. What does it mean for two rules to conflict?

Another significant advantage of having the names the same is that when/if things get added to an apt repository in the future the new rosdep key is just removing the '-pip' extension which is easier for maintainers and clear what the substitute should be.

Does this mean PyPI rules should be removed when platform specific packages become available? foobar-pip and python3-foobar should never exist at the same time? What happens if they do?

[sloretz]

@tfoote friendly ping :)

[tfoote]

I don't think I understand this part. What does it mean for two rules to conflict?

If I install a pip version of foo I'll get whatever is in pypi and it will mask the underlying apt version. If they were the same version that's not problem. But generally pip is far ahead of the apt versions.

Does this mean PyPI rules should be removed when platform specific packages become available? foobar-pip and python3-foobar should never exist at the same time? What happens if they do?

Generally yes

Both of these come down to the fact that pip installations will mask the debian based installations. If I depend on foo as installed via apt. I know what version I'm getting. If someone else pip installs foo they'll get whatever version is latest in pypi. This may be a major new version, with newer upgraded APIs and possibly changed dependencies etc. That's not really a problem if I want the latest version and ask for it. But if foo is a library that other people are depending on the pypi version is rarely 100% compatible so other packages relying on foo we be potentially broken with the unbenownst upgrade.

This is particularly pernicious because as a developer I can assert that I need debian package foo at version X.Y.Z and can have my installation scripts check that the debian package is installed at the right version. However, rosdep will assert that the debian package is installed correctly. However when I try to import the package I'll get the pip version.

Even more confusing is if I have everything working. And then I add a new package to my workspace that you then use rosdep to resolve the pip version will then potentially break an already running system as a side effect of installing.

So at the high level the pip rules should only be added for targets or platforms for which there does not exist an apt or other default installer version.

We have the convention of having the -pip rules to be clear about what's pip only. And what we should be doing is when we add an apt rule we should null out the pip rule for that and future versions. (We don't have automation for this so it often doesn't happen.) We do allow you to add pip resolutions for apt packages on older platforms where the apt package is not available and the pip one can backfill where there's no backports, because those won't conflict. But as a rule the rosdep keys should be defining the canonical version of the package that we are standardizing on using on that platform. And developers can have an expectation that if they rely on that key, they will have what it promises is available.

So this is why I want the same prefix so that when reviewing pull requests these conflicts can be noticed when reviewing PRs more easily.

[sloretz]

So this is why I want the same prefix so that when reviewing pull requests these conflicts can be noticed when reviewing PRs more easily.

@tfoote Updated PR to require python3- prefix and require old -pip keys to be deleted in c3935b7

[clalancette]

One really minor nit. Up to you whether to accept it or not. Approving anyway.

[nuclearsandwich]

Seeing that @tfoote's requested changes have been addressed and that other approvals are all in. This looks good to me too.

[sloretz]

@tfoote would you be willing to check that the changes match your comments before I merge it?

Review comments addressed

[tfoote]

Thanks looks good to me.