Python dependencies updated version not supported by the python version

[ReenigneArcher]

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us what version of Renovate you run.

No response

If you're self-hosting Renovate, select which platform you are using.

None

Was this something which used to work for you, and then stopped?

I never saw this working

Wanted end result.

I am investigating migrating from Dependabot to Renovate.

One "regression" I see is that when updating python dependencies, Renovate will upgrade a dependency pinned to a specific version of Python, to a version of the dependency which is not actually supported by that version of python.

An example in a requirements.txt file

backports.datetime-timestamp==1.2;python_version<"3.8"
backports.datetime-timestamp==1.3.0;python_version>="3.8"

The 1.2 version is updated to 1.3.0, but as you can see on pypi (https://pypi.org/project/backports.datetime-timestamp/1.3.0/) this version is not compatible with < Python 3.8.

What you tried so far.

Searched docs, issues, discussions, but might have missed something. Found #12727 but my understanding is that the python version is ignored?

Is there any config option or workaround to handle this?

Relevant debug logs

Logs

    "pip_requirements": [
      {
        "deps": [
          {
            "depName": "backports.datetime-timestamp",
            "currentValue": "==1.2",
            "datasource": "pypi",
            "currentVersion": "1.2",
            "updates": [
              {
                "bucket": "non-major",
                "newVersion": "1.3.0",
                "newValue": "==1.3.0",
                "releaseTimestamp": "2023-06-24T00:33:12.000Z",
                "newMajor": 1,
                "newMinor": 3,
                "updateType": "minor",
                "isRange": true,
                "branchName": "renovate/backports.datetime-timestamp-1.x"
              }
            ],
            "packageName": "backports.datetime-timestamp",
            "versioning": "pep440",
            "warnings": [],
            "sourceUrl": "https://github.com/jaraco/backports.datetime_timestamp",
            "registryUrl": "https://pypi.org/pypi",
            "isSingleVersion": true,
            "fixedVersion": "1.2"
          },

[ReenigneArcher]

I'm not familiar enough with this code base (otherwise I would put up a PR), but the requires_python field can be pulled from the pypi api https://pypi.org/pypi/${depName}/json.

I think you could do something along these lines.

1. get all versions from api
2. sort versions with newest first
3. loop over each version until requires_python meets all the environment markers for the dependency

Note: The releases key in the api response is not in numerical order. For example it could be ordered like this:

• 4.1.0
• 4.10.0
• 4.2.0

[viceice]

try this
https://docs.renovatebot.com/configuration-options/#constraintsfiltering

[ReenigneArcher]

Thanks for the response, but I think I get the same result.

I should add that I'm still in the "onboarding" stage and have not implemented anything yet (I want to ensure things are ironed out before dumping into 30 production repos)

I have my org level config in our .github repo:

https://github.com/LizardByte/.github/blob/master/renovate-config.json which really just points to https://github.com/LizardByte/.github/blob/master/renovate-config.json5 (onboarding didn't seem to pick up the json5)

I re-ran renovate from the mend.io dashboard, although it didn't update the onboarding PR so that may have something to do with it.

DEBUG: Onboarding cache is valid. Repo is not onboarded
DEBUG: Repo is not onboarded
DEBUG: getBranchPr(renovate/configure)
DEBUG: findPr(renovate/configure, undefined, open)
DEBUG: Found PR #39
DEBUG: Onboarding PR already exists
DEBUG: Skip processing since the onboarding branch is up to date and default branch has not changed

This is the onboarding PR: LizardByte/python-plexapi-backport#39

Just to check, I initiated an onboarding PR for another repo where this would be an issue, and it's the same (ignoring python version from requirements.txt files) LizardByte/Themerr-plex#195

Also, side question. Is this section actually considering my config, or is it just considering the default config? I would actually expect with my config that it should say build(deps): bump <dependencyName> to <dependencyVersion>

Seems, new PRs say chore(deps): bump ..., why is it ignoring the semanticCommitType config option?

[github-actions[bot]]

Hi there,
Please stick to one topic/question per Discussion. Start a new discussion per topic.
This prevents future users from finding this discussion and getting confused by multiple questions and answers. One question and answer per Discussion works best.
For example:

• Don't create a discussion with multiple questions requiring multiple answers
• If you got an answer to your first question, don't change topic like "Can I also ask you how I might..." which would then require a different answer. Mark the current discussion as answered and open a new one.

Thanks, the Renovate team

[rarkins]

If you restrict every dependency to python_version<"3.8", and you want Renovate to ignore updates which are incompatible, then eventually you'll have none of your dependencies getting updates but meanwhile you'll be out of date on all of them. So ignoring updates due to incompatibility does not seem like a sensible default without some other capability to ensure you don't end up out of date and not knowing it.

Possible approaches:

• Update by default like today, but also bump the Python constraint at the same time, or
• Do not update incompatible dependencies, but instead have a different PR which bumps all python constraints to latest. After such a PR is merged, dependency updates would follow

[ReenigneArcher]

If you restrict every dependency to python_version<"3.8"

I don't... This project has known compatibility with (and maybe a few lower than 3.7 but can't easily test in CI)

2.7
3.7
3.8
3.9
3.10
3.11
3.12

So requirements.txt looks like this.

requests;python_version>="3.8"

backports.datetime-timestamp==1.2;python_version<"3.8"
backports.datetime-timestamp==1.3.0;python_version>="3.8"
backports.shutil_which==3.5.2;python_version<"3.3"
cached-property==1.5.2;python_version<"3.8"
configparser==4.0.2;python_version<"3"
future==0.18.3  # needed for all versions since patched setup.py uses it
pathlib2==2.3.7.post1;python_version<"3.4"
requests==2.27.1;python_version<"3"
six==1.16.0
typing;python_version<"3.5"
qualname==0.1.0;python_version<"3.3"

and a more complex project

flake8==3.9.2;python_version<"3.6"
flake8==5.0.4;python_version>="3.6"
m2r2==0.3.2;python_version<"3.7"
m2r2==0.3.2;python_version>="3.7"
numpydoc==0.9.2;python_version<"3.5"
numpydoc==1.1.0;python_version>="3.5" and python_version<"3.7"
numpydoc==1.6.0;python_version>="3.7"
plexapi-backport[alert]==4.15.4;python_version<"3.8"
plexapi[alert]==4.15.4;python_version>="3.8"
pytest==4.6.11;python_version<"3.5"
pytest==6.1.2;python_version>="3.5" and python_version<"3.6"
pytest==7.0.1;python_version>="3.6" and python_version<"3.7"
pytest==7.4.2;python_version>="3.7"
pytest-cov==2.12.1;python_version<"3.6"
pytest-cov==4.0.0;python_version>="3.6" and python_version<"3.7"
pytest-cov==4.1.0;python_version>="3.7"
rstcheck==3.5.0;python_version<"3.7"
rstcheck==6.1.2;python_version>="3.7" and python_version<"3.8"
rstcheck==6.2.0;python_version>="3.8"
Sphinx==1.8.6;python_version<"3.5"
Sphinx==3.5.4;python_version>="3.5" and python_version<"3.6"
Sphinx==5.3.0;python_version>="3.6" and python_version<"3.8"
Sphinx==7.1.2;python_version>="3.8" and python_version<"3.9"
Sphinx==7.2.6;python_version>="3.9"
sphinx-rtd-theme==1.3.0
sphinx-tabs==1.1.13;python_version<"3.6"
sphinx-tabs==3.3.1;python_version>="3.6" and python_version<"3.7"
sphinx-tabs==3.4.1;python_version>="3.7"
tqdm==4.64.1;python_version<"3.7"
tqdm==4.66.1;python_version>="3.7"

I pin dependencies so CI tests do not randomly fail due to outside circumstances.

[rarkins]

Ok so you're running years-old out of date dependencies and don't want to change that for most of them?

What's the purpose of having mixed and/or inconsistent python dependencies? Eg some <3.3, some >=3.8, etc

Also when you use restrictions like <3.3 what is the outcome you're after? Eg you want a version that supports every python version ever since the beginning of time? Or you want to avoid versions which support >=3.3?

[ReenigneArcher]

Unfortunately these projects are tied to Plex, which is stuck on Python 2.7 for their plugin eco system.

What's the purpose of having mixed and/or inconsistent python dependencies? Eg some <3.3, some >=3.8, etc

In some cases a feature was added in Python 3.3, so I have to install a backport for it on versions older than that. qualname==0.1.0;python_version<"3.3" is an example of that.

This one numpydoc==1.1.0;python_version>="3.5" and python_version<"3.7" is because 1.1.0 is the newest version that supports Python 3.5 and 3.6.

[ReenigneArcher]

Also when you use restrictions like <3.3 what is the outcome you're after? Eg you want a version that supports every python version ever since the beginning of time? Or you want to avoid versions which support >=3.3?

Only going back to 2.7.

Besides a setup.py file I don't know if there is a way to specify the supported python versions for the project. I don't think you can rely on or expect that file to exist though.

[rarkins]

I will convert this to a feature request issue once we have a better idea of the logic to use. Here is a draft:

---

Support python_requirements fields in requirements.txt files.

Simple example:

backports.datetime-timestamp==1.2;python_version<"3.8"
backports.datetime-timestamp==1.3.0;python_version>="3.8"

The above means "if you're using python before 3.8, install version 1.2, otherwise if you're using python 3.8 or later then use version 1.3.0.

In this case the top line should not be updated unless somehow there was a newer version of the package which still supports <3.8 python. Most likely the only one of those to ever be updated is the second line, if e.g. 1.3.1 or 1.4.0 were to be released.

v1.2 requires Python >= 2.7

v1.3.0 requires Python >= 3.8

This likely can reuse some of the logic in constraintsFiltering but might need a new value with behavior different than strict because in the above case the user is not after a full subset. For example >=2.7 is not a subset of <3.8. This algorithm remains to be decided.

---

To use the above example, what should happen if there's a version 1.2.1 released which require Python >=3.7? Technically that has an overlap with <3.8, but in reality you probably want Python 2.7 support. I think the cause of this confusion is that you are not using lower bounded ranges, e.g. you really should say >=2.7 <3.8 and then there'd be no confusion. But assuming that it's common to specify unbound ranges like you have, what should Renovate do?

[ReenigneArcher]

Awesome, I really appreciate this being considered!

you really should say >=2.7 <3.8

I have no problem updating my requirements files to include the lower bound, but as you say that is probably not too common. Just as a maximum version of python of python would probably not be set when you say python_version >= 3.8

what should Renovate do?

Maybe something like this. Probably set variables for a min and max version of python. If min is null, then only check the upper bound. i.e. For <3.8, Get the latest version that supports 3.7.*

Opposite for if max is null... only check the lower bound. i.e For >3.8, Get the version that supports 3.9.*

If both are set, then check if the min and max are satisfied.

I think you can probably ignore the patch levels for python version. I'm not sure that is even allowed to be included in the python_requires field.

[rarkins]

So in that case if 1.2.1 was released and only supported >=3.7, you'd consider that a valid upgrade for the first line above, even though it's dropping Python <3.7 support including 2.7?

What I suspect is:

• If we check for the whole range then you probably will miss out on upgrades which you're expecting because we're too strict
• If we are not strict, you'll probably get upgrades which "narrow" the range undesirably

Ultimately the cause of this is bad python tooling though, so there's not much we can do except pick a sensible default (e.g. non strict) and let people opt into more strict

[ReenigneArcher]

So in that case if 1.2.1 was released and only supported >=3.7, you'd consider that a valid upgrade for the first line above, even though it's dropping Python <3.7 support including 2.7?

Yes.

And yea it's not a perfect packaging system... and many developers forget to update the python_requires field when they drop support for a version. So there's always going to be some edge cases.

[lmilbaum]

I'm hitting a similar issue but not exactly the same. The requirements.txt file is used for installing packages in a container image. The expectation is that the identified versions should take into consideration the python version used in the container image during build time.

[rarkins]

Converted to feature request issue: #26310