Renovate CE: Autodiscover includes all repos in failure modes

[sburf-rokt]

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us what version of Renovate you run.

ghcr.io/mend/renovate-ce:6.2.0

If you're self-hosting Renovate, select which platform you are using.

github.com

Was this something which used to work for you, and then stopped?

I am trying to get this working for the first time

Describe the problem

When I launched our Renovate prod instance it sent onboarding PRs for all of our repos. I expected that autodiscoverTopics: ["renovate"] would have prevented this.

Our dev instance and prod instance are different GitHub Apps. The root cause appears to be that when no repos match the filtering, autodiscoverRepositories() returns without clearing the repositories list. The log messages make it pretty clear what happened.

The effective config, from the first "Combined config" log, is:

Combined config

{
  "config": {
    "autodiscover": true,
    "autodiscoverTopics": [
      "renovate"
    ],
    "binarySource": "install",
    "dependencyDashboardFooter": " - [ ] <!-- manual job -->Check this box to trigger a request for Renovate to run again on this repository",
    "endpoint": "https://api.github.com/",
    "gitAuthor": "...",
    "onboarding": true,
    "onboardingConfig": {
      "$schema": "https://docs.renovatebot.com/renovate-schema.json",
      "extends": [ "config:recommended" ]
    },
    "platform": "github",
    "repositories": [
        ".../..."
    ],
    "requireConfig": "required",
    "token": "***********",
    "username": "..."
  },
  "hostname": "...",
  "level": 20,
  "logContext": "...",
  "name": "renovate",
  "pid": 257,
  "time": "2023-10-17T00:28:59.168Z",
  "v": 0
}

The log where things diverge from dev is:

2023-10-17T00:28:16.816Z	Autodiscovering GitHub repositories
2023-10-17T00:28:21.125Z	The account associated with your token does not have access to any repos

versus the expected:

Autodiscovering GitHub repositories
Configured repository is in not in autodiscover list

I believe that this is because in autodiscoverRepositories() the early exits do not clear config.repositories when there is autodiscovery filtering but there is a problem.

Here's the normal exit for autodiscoverRepositories():

  return { ...config, repositories: discovered };

Here's the exit that I encountered:

  if (!discovered?.length) {
    // Soft fail (no error thrown) if no accessible repositories
    logger.debug(
      'The account associated with your token does not have access to any repos'
    );
    return config;
  }

Discovered has no length because no repos are tagged with the production topic renovate yet.

In the early exit, config.repositories is populated even though they don't match my autodiscoverTopics.

There are several early exits in autodiscoverRepositories() where this can occur. I believe that config.repositories should be cleared at the start of autodiscoverRepositories() to fix this.

Relevant debug logs

Logs

2023-10-17T00:28:13.639Z	Running Renovate job
2023-10-17T00:28:13.640Z	Executing Renovate run command
2023-10-17T00:28:14.564Z	Using RE2 as regex engine
2023-10-17T00:28:16.659Z	Parsing configs
2023-10-17T00:28:16.659Z	Checking for config file in /usr/src/app/config.js
2023-10-17T00:28:16.674Z	Detected config in env RENOVATE_CONFIG
2023-10-17T00:28:16.677Z	File config
2023-10-17T00:28:16.677Z	CLI config
2023-10-17T00:28:16.677Z	Env config
2023-10-17T00:28:16.678Z	Combined config
2023-10-17T00:28:16.678Z	Adding trailing slash to endpoint
2023-10-17T00:28:16.722Z	Found valid git version: 2.42.0
2023-10-17T00:28:16.724Z	Platform config
2023-10-17T00:28:16.724Z	Using configured gitAuthor (...)
2023-10-17T00:28:16.729Z	Adding token authentication for api.github.com to hostRules
2023-10-17T00:28:16.731Z	Using baseDir: /tmp/renovate
2023-10-17T00:28:16.734Z	Using cacheDir: /tmp/renovate/cache
2023-10-17T00:28:16.735Z	Using containerbaseDir: /tmp/renovate/cache/containerbase
2023-10-17T00:28:16.747Z	Initializing Renovate internal cache into /tmp/renovate/cache/renovate/renovate-cache-v1
2023-10-17T00:28:16.748Z	Commits limit = null
2023-10-17T00:28:16.748Z	Setting global hostRules
2023-10-17T00:28:16.750Z	Adding token authentication for api.github.com to hostRules
2023-10-17T00:28:16.750Z	validatePresets()
2023-10-17T00:28:16.816Z	Autodiscovering GitHub repositories
2023-10-17T00:28:21.125Z	The account associated with your token does not have access to any repos
2023-10-17T00:28:21.128Z	Reinitializing hostRules for repo
2023-10-17T00:28:21.128Z	Clearing hostRules
2023-10-17T00:28:21.128Z	Adding token authentication for api.github.com to hostRules
2023-10-17T00:28:21.130Z	Repository started
2023-10-17T00:28:21.130Z	Using localDir: /tmp/renovate/repos/github/.../...
2023-10-17T00:28:21.130Z	PackageFiles.clear() - Package files deleted
2023-10-17T00:28:21.136Z	initRepo(".../...")
2023-10-17T00:28:21.416Z	.../... default branch = master
2023-10-17T00:28:21.417Z	Using app token for git init
2023-10-17T00:28:21.946Z	Resetting npmrc

Have you created a minimal reproduction repository?

I have explained in the description why a minimal reproduction is impossible

[rarkins]

1. Is this a problem with all of Autodiscover (as the title implies) or something specific to the autodiscoverTopics feature only?
2. Can you clarify your "diverged from dev" part - I don't understand what you're referring to there. "does not have access to any repos" sounds like what to expect, no?

[rarkins]

When I set autodiscoverTopics, same thing:

DEBUG: Autodiscovering GitHub repositories
DEBUG: The account associated with your token does not have access to any repos
DEBUG: Checking file package cache for expired items
DEBUG: Deleted 0 of 82 file cached entries in 101ms
DEBUG: Renovate exiting

[rarkins]

Meanwhile with no filter or topics:

DEBUG: Autodiscovering GitHub repositories
 INFO: Autodiscovered repositories
       "length": 1755,

Sorry but it seems your assertions are wrong. We can add increased logging through to improve the observability

[sburf-rokt]

I think that the difference might be that you were using the CLI, where the issue is not evident unless you specify a repo? The Renovate CE scheduler seems to provide a repo for each instance of Renovate that it runs.

Specifying a repo on the command line is necessary for demonstrating this bug via the CLI.

Here's a repro with a bogus topic, resulting in an invite being sent for the repo specified on the command line:

renovate \
   --dry-run \
   --autodiscover true \
   --autodiscover-topics doesnotexist \
   --endpoint https://api.github.com \
   --onboarding true \
   --platform github \
   --require-config=required \
   --token "${TOK}" \
   --log-file-level 60 \
   ORG/repo

 WARN: cli config dryRun property has been changed to full
 INFO: Repository started (repository=ORG/repo)
       "renovateVersion": "36.42.0"
 INFO: Dependency extraction complete (repository=ORG/repo, baseBranch=main)
       "stats": {
         "managers": {
           "buildkite": {"fileCount": 2, "depCount": 7},
           "dockerfile": {"fileCount": 2, "depCount": 3},
           "gomod": {"fileCount": 1, "depCount": 53},
           "helm-values": {"fileCount": 1, "depCount": 1},
           "terraform": {"fileCount": 1, "depCount": 4}
         },
         "total": {"fileCount": 7, "depCount": 68}
       }
 INFO: DRY-RUN: Would remove edited/blocked PR comment in PR #22 (repository=ORG/repo, branch=renovate/ecr-2.x)
 INFO: DRY-RUN: Would commit files to branch renovate/ecr-2.x (repository=ORG/repo, branch=renovate/ecr-2.x)
 INFO: DRY-RUN: Would update PR #22 (repository=ORG/repo, branch=renovate/ecr-2.x)
 INFO: DRY-RUN: Would ensure Dependency Dashboard (repository=ORG/repo)
       "title": "Dependency Dashboard"
 INFO: DRY-RUN: Would save repository cache. (repository=ORG/repo)
 INFO: Repository finished (repository=ORG/repo)
       "cloned": true,
       "durationMs": 20054

[rarkins]

Sorry, I missed that part.

This is the wrong repo for a CE problem with autodiscover, so I will close this and you'll need to open a new https://github.com/mend/renovate-ce-ee/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc

Autodiscover is different in CE/EE and does not support all the options.