Ensure */extern/__init__.py is in sync with */_vendor/vendored.txt

[abravalheri]

Sometimes I have the feeling that the vendored packages, _vendor/vendored.txt and extern/__init__.py drift apart. This is an attempt to minimise that.

Please feel free to close if you don't think this is relevant.

• Note: I haven't run tox -e vendor in this PR.

Summary of changes

• Added a small cog script that attempts to derive the *.extern.names from the */_vendor/vendored.txt file
• Added support function in tools/vendored.py
• Modified tox -e vendor to automatically run cog
• Added tox -e check-extern.
• Added more_itertools to pkg_resources/_vendor/vendored.txt? (maybe it was missing?)

Closes

Pull Request Checklist

• Changes have tests
• News fragment added in newsfragments/.
  (See documentation for details)

[abravalheri]

@jaraco, zipp was not in this pkg_resources/extern/__init__.py, but the vendored.txt file says:

# required for importlib_resources on older Pythons
zipp==3.7.0

Maybe that was a bug? Or is "older Pythons" here referring to Python < 3.7? In that case maybe we can remove it?

[jaraco]

I'm unsure what older Pythons means, as it's dependent on the version of importlib_resources that's resolved. I would keep a version compatible with whatever importlib_resources declares.

[abravalheri]

This was not listed. The vendored.txt file says the following:

# required for platformdirs on Python < 3.8
typing_extensions==4.4.0

But now we removed support to Python < 3.8, so I guess we can remove this vendored dependency?

[abravalheri]

Minor change: automatic reordering, just so it matches vendored.txt.

[abravalheri]

I added this here because it was listed in pkg_resources.extern.names.
The version corresponds to the version that already exists in the pkg_resources/_vendor the directory... But that is different from the version in setuptools/_vendor/vendored.txt. Maybe we should synchronise these versions so that it makes the life easier for downstream packagers "de-vendoring" setuptools?

[jaraco]

Thanks for taking care of this. I've never loved the two-step manual sync aspect of the vendoring design. I've been thinking we may be able to simplify it by merely adding _vendor to sys.path (and eliminating extern altogether), but I haven't experimented with that and I'm unsure what dragons might lurk in that approach.

[jaraco]

I'm unsure what older Pythons means, as it's dependent on the version of importlib_resources that's resolved. I would keep a version compatible with whatever importlib_resources declares.

[Avasam]

I've been thinking we may be able to simplify it by merely adding _vendor to sys.path (and eliminating extern altogether)

@jaraco Eliminating extern in favor of importing from _vendor (or making it so import vendored_module imports _vendor.vendored_module) would also solve type imports issues (like #4242) for type-checkers.

[abravalheri]

I think we can go with this one for now and any improvements can follow up. I also opened other issues for the comments we left.

I've been thinking we may be able to simplify it by merely adding _vendor to sys.path (and eliminating extern altogether), but I haven't experimented with that and I'm unsure what dragons might lurk in that approach.

Yeah, that would be good. The obvious drawback is the dependency clash if other packages installed in the same environment require conflicting versions.

A different approach (as referred by @Avasam) would be to eliminate the MetaPathFinder (and therefore the extern), but still explicitly import from setuptools._vendor. That would simply a little while avoiding the version clash. We still would have to keep the patching for absolute imports of all modules installed inside of _vendor.