Is Renovate using Poetry >=1.8.0 consistently?

[kkom]

What would you like help with?

I think I found a bug

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

No response

Please tell us more about your question or problem

We're seeing intermittent errors like this:

The Poetry configuration is invalid:
  - Additional properties are not allowed ('package-mode' was unexpected)

This package-mode property was added in python-poetry/poetry#8650 and released in Poetry 1.8.0: https://github.com/python-poetry/poetry/releases/tag/1.8.0

It seems like Renovate is sometimes running with an old version of Poetry and sometimes with a new one.

Shall we stop setting this property for now, or are you able to ensure that you're using Poetry >=1.8.0 when creating PRs?

Logs (if relevant)

No response

[github-actions[bot]]

Hi there,

This issue or discussion is missing some logs, making it difficult or impossible to help you. Depending on which situation applies follow one, some or all of these instructions.

No logs at all

If you haven't posted any log yet, we need you to find and copy/paste the log into the issue template.

Finding logs on hosted app

Select me to read instructions

If you use the Mend Renovate app (GitHub):

1. Log in to the Mend Developer Portal
2. Navigate to the correct organization and repository
3. Locate the appropriate log (it may not always be the latest one)
4. Copy/paste the log contents
5. Follow the steps in the formatting your logs section

Finding logs when self-hosting

Select me to read instructions

Read the Renovate docs, troubleshooting, self-hosted to learn how to find the logs.

Insufficient logs

Select me to read instructions

If you already gave us a log, and the Renovate team said it's not enough, then follow the instructions from the No logs at all section.

Formatting your logs

Select me to read instructions

Please put your logs in a <details> and <summary> element like this:

<details><summary>Select me to see logs</summary>

```
Copy/paste your logs here, between the starting and ending backticks
```

</details>

If you feel the logs are too large to paste here, please use a service like GitHub Gist and paste the link here.

Good luck,

The Renovate team

[rarkins]

In theory Renovate should use the latest version of tools unless it thinks the project is configured otherwise

[rarkins]

I think if Renovate cannot determine a version via a proper constraint it should fall back to the default behavior, which I think is to take the latest Poetry version.

What do you mean by "determine a version via a proper constraint"?

Do you mean that unless there's a constraint in the pyproject (does such a thing exist?) then ignore the lock file because it can't be relied on? Or in other words, always use the latest even though that's really non-ideal?

What we should IMO do is improve the documentation and maybe the logging so that folks that run into this can more quickly understand what's going on.

We should definitely make constraint logging more verbose wherever needed

[fgreinacher]

Do you mean that unless there's a constraint in the pyproject (does such a thing exist?) then ignore the lock file because it can't be relied on? Or in other words, always use the latest even though that's really non-ideal?

I think the only thing that is meant to be relied upon is the build-system section in the pyproject file, see https://python-poetry.org/docs/pyproject/#poetry-and-pep-517:

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Renovate does already support this:

renovate/lib/modules/manager/poetry/artifacts.ts

Lines 76 to 83 in e34248b

	const { val: pyprojectPoetryConstraint } = Result.parse(
	pyProjectContent,
	PoetrySchemaToml.transform(({ poetryRequirement }) => poetryRequirement),
	).unwrap();
	if (pyprojectPoetryConstraint) {
	logger.debug('Using poetry version from pyproject.toml');
	return pyprojectPoetryConstraint;
	}

But unfortunately the lockfile is considered first, so maybe we could improve by changing only the order as mentioned in #28688 (reply in thread).

[rarkins]

What's going to happen here when poetry releases a 2.0.0 with breaking changes?

[fgreinacher]

How is that handled for other package managers like Yarn, NPM, or NuGet? Isn't the default to go with latest?

[rarkins]

If we cannot find any indicators to tell us which version to use, then we use latest LTS/stable

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this. Discussions without reproductions are less likely to be converted to Issues.

To get started, please read our guide on creating a minimal reproduction.

Good luck,

The Renovate team

[fgreinacher]

So, I have investigated this a bit and I still believe that interpreting the version from the lockfile is not a good choice, for the following reasons:

1. The version number in the lockfile is only updated if the dependency tree changes, even when running poetry lock explicitly.
2. In general Poetry version numbers are not something to rely on, see https://python-poetry.org/docs/faq#what-kind-of-versioning-scheme-does-poetry-use-for-itself.
3. There is a well-defined option to state the minimum Poetry version, see https://python-poetry.org/docs/pyproject/#poetry-and-pep-517

In my opinion we should go forward like this:

1. Remove the code path that interprets the version in the lockfile header. If that is too big of a change we could put it behind a configuration option.
2. Add a new code path to define a >=1.8.0 constraint when we detect the new properties in the project file, making this "just work"
3. Add a chapter to the documentation that explains our logic and strongly suggest people to pin their Poetry versions as described in https://python-poetry.org/docs/pyproject/#poetry-and-pep-517

@rarkins had the concern that using Poetry latest might cause a lot of lockfile churn due to the version in the lockfile header. As mentioned above that won't happen because the version is only updated when the dependency tree changes.

[HonkingGoose]

We're working on a breaking major release anyways. So maybe your big behavior change can go in that? We can then use the changelog commentary to explain the how and why of the big change.

Documenting that users should pin their dependencies seems good to me.

[MattKobayashi]

Just wanted to add that I've encountered this same issue despite poetry-core being pinned to version 1.9.0. I don't think pinning the Poetry version is a reliable solution to the issue.

[kkom]

@rarkins @HonkingGoose I have a related issue – this time to do with choosing a Python version.

(I'll start with a thread in this discussion, but I'm happy to create a new one if you prefer me to.)

In our project we pin Python version down to a patch, so our pyproject.toml file looks like this:

[tool.poetry.dependencies]
...
python = "3.12.3"
...

It may not be the usual way of doing things, but we do it mainly out of convenience. It keeps the Renovate configuration simple as we pin all other dependencies. We also have .tool-versions / Dockerfile files where we need Python to be pinned to patch, as patch releases of Python occasionally do cause issues, e.g. pydantic/pydantic#9607 right now.

However, when a new patch version of Python comes out, Renovate will immediate use it, leading to problems updating artifacts:

Wdyt about running analogous version inference logic for Python, as is currently done for Poetry? (See #28688 (reply in thread) )

I know that @fgreinacher is explicitly suggesting to drop that Poetry behaviour in #28688 (comment), so I'm happy to work on updating our pyproject.toml file to allow Python ranges, while keeping it pinned to patches elsewhere. But would love to hear some comments on that from you before I do it.

[rarkins]

Please copy/paste the relevant debug logs for when a commit is made