pip-compile can fail artifact update check with multiple inputs

[not7cd]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

CircleCI

Please tell us more about your question or problem

Problem

Renovate will fail a check for artifact update. Even though it eventually succeeds in this update by running pip-compile more than once.

Description

Given this requirement file hierarchy:

       graph LR
         requirements-jupyter.txt[[requirements-jupyter.txt]]
         requirements.txt[[requirements.txt]]
         irrelevant.txt[[irrelevant.txt]]
         requirements-shared.in --> irrelevant.txt
         requirements-jupyter.in --> requirements-jupyter.txt
         requirements.txt --> requirements-jupyter.in
         requirements-shared.in --> requirements.txt
         requirements.in --> requirements.txt 
         

Loading

We can expect Renovate to run pip-compile more than once on a lock file. This is because all descending lock files are added to a package file, as introduced in #28052.

This indeed happens, but in an order that throws artifact update error for requirements-jupyter.txt.

DEBUG: updateArtifacts for nonUpdatedPackageFiles
# ...
DEBUG: pipCompile.updateArtifacts(requirements-jupyter.in->["requirements-jupyter.txt"]) 
DEBUG: pip-compile: Failed to run command
       "err": {
         "cmd": "/bin/sh -c pip-compile --all-extras --no-emit-index-url --output-file=requirements-jupyter.txt requirements-jupyter.in --upgrade-package=foo==1.2.3",
         "stderr": "  ERROR: Cannot install foo==1.2.3 because these package versions have conflicting dependencies. ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts",
         "stdout": "",
         "options": {
             ...
         },
         "exitCode": 1,
         "name": "ExecError",
         "message": "..."
       }
DEBUG: pipCompile.updateArtifacts(requirements-shared.in->["irrelevant.txt","requirements.txt","requirements-jupyter.txt"]) 
DEBUG: pipCompile.updateArtifacts(requirements.in->["requirements.txt","requirements-jupyter.txt"]) 
# ...

The pipCompile.updateArtifacts(requirements-jupyter.in->["requirements-jupyter.txt"]) will fail because it depends on requirements.txt which is not yet compiled. It also all happens within the context of updateArtifacts for nonUpdatedPackageFiles

Possible fix

Keep the package file order based on topological sort during artifactUpdate. I don't remember if this order was ever respected.

Alternatively, remove artifactErrors if it succeds in another artifactUpdate execution. This must happen outside pip-compile module.

The fix may require looking into the whole updateArtifacts section in get-updated.

renovate/lib/workers/repository/update/branch/get-updated.ts

Lines 298 to 394 in 7178da3

	for (const packageFile of updatedPackageFiles) {
	const updatedDeps = packageFileUpdatedDeps[packageFile.path];
	const managers = packageFileManagers[packageFile.path];
	if (is.nonEmptySet(managers)) {
	for (const manager of managers) {
	const results = await managerUpdateArtifacts(manager, {
	packageFileName: packageFile.path,
	updatedDeps,
	// TODO #22198
	newPackageFileContent: packageFile.contents!.toString(),
	config: patchConfigForArtifactsUpdate(
	config,
	manager,
	packageFile.path,
	),
	});
	processUpdateArtifactResults(
	results,
	updatedArtifacts,
	artifactErrors,
	artifactNotices,
	);
	}
	}
	}
	const nonUpdatedPackageFiles: FileAddition[] = Object.keys(
	nonUpdatedFileContents,
	).map((name) => ({
	type: 'addition',
	path: name,
	contents: nonUpdatedFileContents[name],
	}));
	// istanbul ignore if
	if (is.nonEmptyArray(nonUpdatedPackageFiles)) {
	logger.debug('updateArtifacts for nonUpdatedPackageFiles');
	}
	for (const packageFile of nonUpdatedPackageFiles) {
	const updatedDeps = packageFileUpdatedDeps[packageFile.path];
	const managers = packageFileManagers[packageFile.path];
	if (is.nonEmptySet(managers)) {
	for (const manager of managers) {
	const results = await managerUpdateArtifacts(manager, {
	packageFileName: packageFile.path,
	updatedDeps,
	// TODO #22198
	newPackageFileContent: packageFile.contents!.toString(),
	config: patchConfigForArtifactsUpdate(
	config,
	manager,
	packageFile.path,
	),
	});
	processUpdateArtifactResults(
	results,
	updatedArtifacts,
	artifactErrors,
	artifactNotices,
	);
	if (is.nonEmptyArray(results)) {
	updatedPackageFiles.push(packageFile);
	}
	}
	}
	}
	if (!reuseExistingBranch) {
	// Only perform lock file maintenance if it's a fresh commit
	// istanbul ignore if
	if (is.nonEmptyArray(lockFileMaintenanceFiles)) {
	logger.debug('updateArtifacts for lockFileMaintenanceFiles');
	}
	for (const packageFileName of lockFileMaintenanceFiles) {
	const managers = packageFileManagers[packageFileName];
	if (is.nonEmptySet(managers)) {
	for (const manager of managers) {
	const contents =
	updatedFileContents[packageFileName] ||
	(await getFile(packageFileName, config.baseBranch));
	const results = await managerUpdateArtifacts(manager, {
	packageFileName,
	updatedDeps: [],
	newPackageFileContent: contents!,
	config: patchConfigForArtifactsUpdate(
	config,
	manager,
	packageFileName,
	),
	});
	processUpdateArtifactResults(
	results,
	updatedArtifacts,
	artifactErrors,
	artifactNotices,
	);
	}
	}
	}
	}

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks

[not7cd]

@mbudnek, did you encounter similar glitches? I think we discussed something, but I'm not sure if there was a conclusion.

[mbudnek]

Yes, I have #28671 open to fix this, but it's been pending for a while now awaiting review.

[viceice]

I didn't had the time to review carefully yet. sorry.