Custom regex-manager is not being executed

[HumanPrinter]

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

37.78.1

If you're self-hosting Renovate, select which platform you are using.

Azure DevOps (dev.azure.com)

What is your question?

I am trying to accomplish a setup where Renovate will detect version updates for tasks in Azure DevOps pipelines (yaml-files), including tasks that we created our self.
To do this, I first created a webservice that can return the available versions for any given task in the plain-text format as described in https://docs.renovatebot.com/modules/datasource/custom/

I then added a custom manager and a custom datasource in my configuration:

    "$schema": "https://docs.renovatebot.com/renovate-schema.json",
    "extends": ["config:best-practices", ":prHourlyLimitNone"],
    "azureWorkItemId": 999,
    "pre-commit": {
      "enabled": true
    },
    "packageRules": [
      {
        "matchUpdateTypes": [
          "minor",
          "patch",
          "pin",
          "digest"
        ],
        "automerge": true
      },
      {
        "matchPackagePatterns": ["pre-commit-hooks"],
        "groupName": ""
      },
      {
        "matchPackageNames": [
          "python"
        ],
        "allowedVersions": "<3.12"
      }
    ],
    "nuget": {
      "registryUrls": [
        "https://api.nuget.org/v3/index.json",
        "https://pkgs.dev.azure.com/<redacted>/_packaging/<redacted>/nuget/v3/index.json"
      ]
    },
    "customManagers": [
      {
        "customType": "regex",
        "fileMatch": [
          "(^|/)pipelines/.*\\.ya?ml"
        ],
        "matchStringsStrategy": "any",
        "matchStrings": [
          "task:\\s+(?<depName>[^\\@]+)\\@(?<currentValue>.*)$"
        ],
        "datasourceTemplate": "custom.azuredevops_pipelinetasks",
        "versioningTemplate": "loose"
      }
    ],
    "customDatasources": {
      "azuredevops_pipelinetasks": {
        "defaultRegistryUrlTemplate": "https://<redacted>/pipelinetasksversions/{{packageName}}.txt",
        "format": "plain"
      }
    }
  }

However, when executing Renovate, the custom regex manager and/or custom dataSource is not being executed. Looking at the debug output, I believe the fileMatch is oke. Using regex101.com I can see no errors in my matchStrings either.

What am I missing?

Logs (if relevant)

Logs

 INFO: Repository started (repository=Git/MyRepo)
       "renovateVersion": "37.78.1"
DEBUG: Using localDir: /tmp/renovate/repos/azure/Git/MyRepo (repository=Git/MyRepo)
DEBUG: PackageFiles.clear() - Package files deleted (repository=Git/MyRepo)
DEBUG: initRepo("Git/MyRepo") (repository=Git/MyRepo)
DEBUG: Repository details (repository=Git/MyRepo)
       **redacted**
DEBUG: Git/MyRepo owner = ?owner? (repository=Git/MyRepo)
DEBUG: Git/MyRepo default branch = main (repository=Git/MyRepo)
DEBUG: Resetting npmrc (repository=Git/MyRepo)
DEBUG: Resetting npmrc (repository=Git/MyRepo)
DEBUG: checkOnboarding() (repository=Git/MyRepo)
DEBUG: isOnboarded() (repository=Git/MyRepo)
DEBUG: getPrList() (repository=Git/MyRepo)
DEBUG: Retrieved Pull Requests count: 6 (repository=Git/MyRepo)
DEBUG: findFile(renovate.json) (repository=Git/MyRepo)
DEBUG: Initializing git repository into /tmp/renovate/repos/azure/Git/MyRepo (repository=Git/MyRepo)
DEBUG: Performing blobless clone (repository=Git/MyRepo)
DEBUG: git clone completed (repository=Git/MyRepo)
       "durationMs": 932
DEBUG: latest repository commit (repository=Git/MyRepo)
       "latestCommit": {
         **redacted**
       }
DEBUG: findFile(renovate.json5) (repository=Git/MyRepo)
DEBUG: findFile(.github/renovate.json) (repository=Git/MyRepo)
DEBUG: Config file exists, fileName: .github/renovate.json (repository=Git/MyRepo)
DEBUG: Repo is onboarded (repository=Git/MyRepo)
DEBUG: Found .github/renovate.json config file (repository=Git/MyRepo)
DEBUG: Repository config (repository=Git/MyRepo)
       "fileName": ".github/renovate.json",
       "config": {
         "$schema": "https://docs.renovatebot.com/renovate-schema.json",
         "extends": ["local>Git/renovate-config"]
       }
DEBUG: migrateAndValidate() (repository=Git/MyRepo)
DEBUG: No config migration necessary (repository=Git/MyRepo)
DEBUG: Setting hostRules from config (repository=Git/MyRepo)
DEBUG: Found repo ignorePaths (repository=Git/MyRepo)
       "ignorePaths": [
         "**/node_modules/**",
         "**/bower_components/**",
         "**/vendor/**",
         "**/examples/**",
         "**/__tests__/**",
         "**/test/**",
         "**/tests/**",
         "**/__fixtures__/**"
       ]
DEBUG: No vulnerability alerts found (repository=Git/MyRepo)
 WARN: findIssue() is not implemented (repository=Git/MyRepo)
DEBUG: No baseBranches (repository=Git/MyRepo)
DEBUG: extract() (repository=Git/MyRepo)
DEBUG: Setting current branch to main (repository=Git/MyRepo)
DEBUG: latest commit (repository=Git/MyRepo)
       "branchName": "main",
       "latestCommitDate": "2023-12-01T08:33:49+00:00"
DEBUG: Using file match: (^|/)tasks/[^/]+\.ya?ml$ for manager ansible (repository=Git/MyRepo)
... ...
!!>> DEBUG: Using file match: (^|/)pipelines/.*\.ya?ml for manager regex (repository=Git/MyRepo)
DEBUG: Matched 1 file(s) for manager dockerfile: docker-image/Dockerfile (repository=Git/MyRepo)
DEBUG: Matched 2 file(s) for manager pip_requirements: docker-image/requirements.txt, scripts/requirements.txt (repository=Git/MyRepo)
DEBUG: Matched 1 file(s) for manager pre-commit: .pre-commit-config.yaml (repository=Git/MyRepo)
!!>> DEBUG: Matched 1 file(s) for manager regex: pipelines/build-and-deploy.yml (repository=Git/MyRepo)**
DEBUG: Found version v4.5.0 (repository=Git/MyRepo)
DEBUG: Found github dependency (repository=https://github.com/pre-commit/pre-commit-hooks)
DEBUG: Found version 37.78.1 (repository=Git/MyRepo)
DEBUG: Found github dependency (repository=https://github.com/renovatebot/pre-commit-hooks)
DEBUG: manager extract durations (ms) (repository=Git/MyRepo)
       "managers": {"dockerfile": 6, "regex": 5, "pre-commit": 10, "pip_requirements": 10}
DEBUG: Found dockerfile package files (repository=Git/MyRepo)
DEBUG: Found pip_requirements package files (repository=Git/MyRepo)
DEBUG: Found pre-commit package files (repository=Git/MyRepo)
DEBUG: Found 4 package file(s) (repository=Git/MyRepo)
 INFO: Dependency extraction complete (repository=Git/MyRepo, baseBranch=main)
       "stats": {
         "managers": {
           "dockerfile": {"fileCount": 1, "depCount": 1},
           "pip_requirements": {"fileCount": 2, "depCount": 3},
           "pre-commit": {"fileCount": 1, "depCount": 6}
         },
         "total": {"fileCount": 4, "depCount": 10}
       }
DEBUG: hostRules: authentication already set for api.github.com (repository=Git/MyRepo)
DEBUG: Falling back to npm semver syntax for allowedVersions (repository=Git/MyRepo)
       "depName": "python"
DEBUG: getDigest(https://index.docker.io, library/python, 3.12.0-slim) (repository=Git/MyRepo)
DEBUG: getManifestResponse(https://index.docker.io, library/python, sha256:eb6d3208444a54418be98f83f1006f6d78ef17144f1cd9eb4e5945d4851af355, head) (repository=Git/MyRepo)
DEBUG: getManifestResponse(https://index.docker.io, library/python, 3.12.0-slim, head) (repository=Git/MyRepo)
DEBUG: Got docker digest sha256:eb6d3208444a54418be98f83f1006f6d78ef17144f1cd9eb4e5945d4851af355 (repository=Git/MyRepo)
DEBUG: PackageFiles.add() - Package file saved for base branch (repository=Git/MyRepo, baseBranch=main)
DEBUG: Package releases lookups complete (repository=Git/MyRepo, baseBranch=main)
DEBUG: branchifyUpgrades (repository=Git/MyRepo)
DEBUG: detectSemanticCommits() (repository=Git/MyRepo)
DEBUG: getCommitMessages (repository=Git/MyRepo)
DEBUG: semanticCommits: detected "unknown" (repository=Git/MyRepo)
DEBUG: semanticCommits: disabled (repository=Git/MyRepo)
DEBUG: 0 flattened updates found:  (repository=Git/MyRepo)
DEBUG: Returning 0 branch(es) (repository=Git/MyRepo)
DEBUG: config.repoIsOnboarded=true (repository=Git/MyRepo)
DEBUG: packageFiles with updates (repository=Git/MyRepo, baseBranch=main)
       "config": {
         "dockerfile": [
           {
             "deps": [
               {
                 "depName": "python",
                 "currentValue": "3.12.0-slim",
                 "currentDigest": "sha256:eb6d3208444a54418be98f83f1006f6d78ef17144f1cd9eb4e5945d4851af355",
                 "replaceString": "python:3.12.0-slim@sha256:eb6d3208444a54418be98f83f1006f6d78ef17144f1cd9eb4e5945d4851af355",
                 "autoReplaceStringTemplate": "{{depName}}{{#if newValue}}:{{newValue}}{{/if}}{{#if newDigest}}@{{newDigest}}{{/if}}",
                 "datasource": "docker",
                 "depType": "final",
                 "updates": [],
                 "packageName": "python",
                 "versioning": "docker",
                 "warnings": [],
                 "registryUrl": "https://index.docker.io",
                 "currentVersion": "3.12.0",
                 "fixedVersion": "3.12.0-slim"
               }
             ],
             "packageFile": "docker-image/Dockerfile"
           }
         ],
         "pip_requirements": [
           {
             "deps": [
               {
                 "depName": "requests",
                 "currentValue": "==2.31.0",
                 "datasource": "pypi",
                 "currentVersion": "2.31.0",
                 "updates": [],
                 "packageName": "requests",
                 "versioning": "pep440",
                 "warnings": [],
                 "sourceUrl": "https://github.com/psf/requests",
                 "registryUrl": "https://pypi.org/pypi",
                 "homepage": "https://requests.readthedocs.io",
                 "changelogUrl": "https://github.com/psf/requests/blob/master/HISTORY.md",
                 "fixedVersion": "2.31.0"
               }
             ],
             "packageFile": "docker-image/requirements.txt"
           },
           {
             "deps": [
               {
                 "depName": "pre-commit",
                 "datasource": "pypi",
                 "updates": [],
                 "packageName": "pre-commit",
                 "versioning": "pep440",
                 "warnings": [],
                 "skipReason": "invalid-value"
               },
               {
                 "depName": "pylint",
                 "datasource": "pypi",
                 "updates": [],
                 "packageName": "pylint",
                 "versioning": "pep440",
                 "warnings": [],
                 "skipReason": "invalid-value"
               }
             ],
             "packageFile": "scripts/requirements.txt"
           }
         ],
         "pre-commit": [
           {
             "deps": [
               {
                 "datasource": "github-tags",
                 "depName": "pre-commit/pre-commit-hooks",
                 "depType": "repository",
                 "packageName": "pre-commit/pre-commit-hooks",
                 "currentValue": "v4.5.0",
                 "updates": [],
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "sourceUrl": "https://github.com/pre-commit/pre-commit-hooks",
                 "registryUrl": "https://github.com",
                 "currentVersion": "v4.5.0",
                 "fixedVersion": "v4.5.0"
               },
               {
                 "datasource": "github-tags",
                 "depName": "renovatebot/pre-commit-hooks",
                 "depType": "repository",
                 "packageName": "renovatebot/pre-commit-hooks",
                 "currentValue": "37.78.1",
                 "updates": [],
                 "versioning": "semver-coerced",
                 "warnings": [],
                 "sourceUrl": "https://github.com/renovatebot/pre-commit-hooks",
                 "registryUrl": "https://github.com",
                 "currentVersion": "37.78.1",
                 "fixedVersion": "37.78.1"
               }
             ],
             "packageFile": ".pre-commit-config.yaml"
           }
         ]
       }
DEBUG: detectSemanticCommits() (repository=Git/MyRepo)
... ...
DEBUG: repository problems (repository=Git/MyRepo)
       "repoProblems": [
         "WARN: findIssue() is not implemented",
         "WARN: ensureIssue() is not implemented"
       ]
DEBUG: Package lookup durations (repository=Git/MyRepo)
       "pypi": {"count": 3, "averageMs": 42, "totalMs": 125, "maximumMs": 77},
       "github-tags": {"count": 6, "averageMs": 2130, "totalMs": 12778, "maximumMs": 2813},
       "docker": {"count": 1, "averageMs": 3097, "totalMs": 3097, "maximumMs": 3097}
DEBUG: dns cache (repository=Git/MyRepo)
       "hosts": []
 INFO: Repository finished (repository=Git/MyRepo)
       "cloned": true,
       "durationMs": 9163```

</details>

[github-actions[bot]]

Hi there,

Get your discussion fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the discussion if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[HumanPrinter]

I've added a minimal reproduction repo to demonstrate the problem: https://dev.azure.com/humanprinter/PublicRepos/_git/RenovateReproduction

The pipeline that holds the output is located here: https://dev.azure.com/humanprinter/PublicRepos/_build/results?buildId=1295&view=results

[rarkins]

Please install the hosted app into your reproduction repo. This will provide a third view of the repo

[HumanPrinter]

Running Renovate using the hosted app does indeed execute the custom datasource. This makes me believe the error might not be in the config after all?
Any clues on how to make it work without using the hosted app? Since we ultimately do not run our workflows/pipelines in GitHub, but in Azure DevOps.

[rarkins]

Try removing everything non-essential in config.js first?

[HumanPrinter]

I finally found the problem. It turned out to be the matchStrings-regex after all. The reason I did not found it at first, was that after migrating by reproduction repo to GitHub, I did not change the config.js (by mistake). It was therefore still looking at the code in my public Azure DevOps repo. Therefore, all changes I made in the GitHub repo did not have any effect.
Anyway, I managed to fix the regex and it now works as expected.