Gitlab renovate-runner project fails with UNABLE_TO_GET_ISSUER_CERT on self hosted gitlab instance

[bl4ko]

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

renovate/renovate:37.203.2-full

If you're self-hosting Renovate, select which platform you are using.

GitLab self-hosted

What is your question?

Hello, on my self hosted gitlab instance I have set up renovate-runner project. Because the self hosted instance uses self signed certificates, I have created my own renovate image following steps in the docs. The Dockerfile looks like this:

FROM renovate/renovate:37.203.2-full

# Changes to the certificate authority require root permissions
USER root

# Copy and install the self signed certificate
COPY src-sub.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

# Change back to the Ubuntu user
USER 1000

# Some tools come with their own certificate authority stores and thus need to trust the self-signed certificate or the entire OS store explicitly.
# This list is _not_ comprehensive and other tools may require further configuration.
#
# Node
ENV NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/src-sub.crt
ENV NODE_OPTIONS=--use-openssl-ca

# Python
RUN pip config set global.cert /etc/ssl/certs/ca-certificates.crt
ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
# OpenSSL
ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

This error won't go away even though the self signed certificate is added into the custom renovate image (shell commands inside image work like curl and git), and env variables are set:

• ENV NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/src-sub.crt
• ENV NODE_OPTIONS=--use-openssl-ca

The only solution, I have found so far and I'm not happy with, was to set env variable NODE_TLS_REJECT_UNAUTHORIZED=0 inside image.

I am wondering what else could be wrong?

Logs (if relevant)

Logs

Running with gitlab-runner 16.9.0 (656c1943)
  on gitlab-shared-runner-gitlab-runner-8659949bc5-rpnk7 LJp81c3sB, system ID: r_KYc3AmSgtjiz
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-csm-runner
Using Kubernetes executor with image harbor.example.com/src-images/renovate:test3 ...
Using attach strategy to execute scripts...
Preparing environment
00:10
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...
Waiting for pod gitlab-csm-runner/runner-ljp81c3sb-project-11-concurrent-1-kkform5i to be running, status is Pending
Waiting for pod gitlab-csm-runner/runner-ljp81c3sb-project-11-concurrent-1-kkform5i to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-csm-runner/runner-ljp81c3sb-project-11-concurrent-1-kkform5i to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-ljp81c3sb-project-11-concurrent-1-kkform5i via gitlab-shared-runner-gitlab-runner-8659949bc5-rpnk7...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/csm/renovate-runner/.git/
Created fresh repository.
Checking out 6cde27cf as detached HEAD (ref is main)...
Skipping Git submodules setup
Restoring cache
00:00
Checking cache for main-renovate-protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
Successfully extracted cache
Executing "step_script" stage of the job script
00:08
$ renovate $RENOVATE_EXTRA_FLAGS
DEBUG: Using RE2 regex engine
DEBUG: Parsing configs
DEBUG: Checking for config file in config.js
DEBUG: No config file found on disk - skipping
DEBUG: Enabling debug logging to renovate-log.ndjson
DEBUG: File config
       "config": {}
DEBUG: CLI config
       "config": {"autodiscover": true}
DEBUG: Env config
       "config": {
         "hostRules": [],
         "repositoryCache": "enabled",
         "dryRun": "full",
         "baseDir": "/builds/csm/renovate-runner/renovate",
         "logFile": "renovate-log.ndjson",
         "logFileLevel": "debug",
         "onboardingConfig": {
           "$schema": "https://docs.renovatebot.com/renovate-schema.json",
           "extends": ["config:recommended"]
         },
         "optimizeForDisabled": true,
         "platform": "gitlab",
         "endpoint": "https://git-csm.example.com/api/v4",
         "token": "***********",
         "gitAuthor": "Renovate Bot <[email protected]>"
       }
DEBUG: Combined config
       "config": {
         "hostRules": [],
         "repositoryCache": "enabled",
         "dryRun": "full",
         "baseDir": "/builds/csm/renovate-runner/renovate",
         "logFile": "renovate-log.ndjson",
         "logFileLevel": "debug",
         "onboardingConfig": {
           "$schema": "https://docs.renovatebot.com/renovate-schema.json",
           "extends": ["config:recommended"]
         },
         "optimizeForDisabled": true,
         "platform": "gitlab",
         "endpoint": "https://git-csm.example.com/api/v4",
         "token": "***********",
         "gitAuthor": "Renovate Bot <[email protected]>",
         "autodiscover": true
       }
DEBUG: Adding trailing slash to endpoint
DEBUG: Found valid git version: 2.43.0
DEBUG: Setting global hostRules
DEBUG: hostRules: authentication already set for git-csm.example.com
DEBUG: GET https://git-csm.example.com/api/v4/version = (code=UNABLE_TO_GET_ISSUER_CERT, statusCode=-1 retryCount=0, duration=54)
DEBUG: Gitlab API error
       "err": {
         "name": "RequestError",
         "code": "UNABLE_TO_GET_ISSUER_CERT",
         "timings": {
           "start": 1708954623657,
           "socket": 1708954623661,
           "lookup": 1708954623665,
           "connect": 1708954623668,
           "error": 1708954623711,
           "phases": {"wait": 4, "dns": 4, "tcp": 3, "total": 54}
         },
         "message": "unable to get issuer certificate",
         "stack": "RequestError: unable to get issuer certificate\n    at ClientRequest.<anonymous> (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/got/dist/source/core/index.js:970:111)\n    at Object.onceWrapper (node:events:632:26)\n    at ClientRequest.emit (node:events:529:35)\n    at ClientRequest.emit (node:domain:489:12)\n    at ClientRequest.origin.emit (/usr/local/renovate/node_modules/.pnpm/@[email protected]/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n    at TLSSocket.socketErrorListener (node:_http_client:501:9)\n    at TLSSocket.emit (node:events:517:28)\n    at TLSSocket.emit (node:domain:489:12)\n    at emitErrorNT (node:internal/streams/destroy:151:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:116:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)\n    at TLSSocket.onConnectSecure (node:_tls_wrap:1659:34)\n    at TLSSocket.emit (node:events:517:28)\n    at TLSSocket.emit (node:domain:489:12)\n    at TLSSocket._finishInit (node:_tls_wrap:1070:8)\n    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:856:12)",
         "options": {
           "headers": {
             "user-agent": "RenovateBot/37.203.2 (https://github.com/renovatebot/renovate)",
             "accept": "application/json",
             "authorization": "***********",
             "accept-encoding": "gzip, deflate, br"
           },
           "url": "https://git-csm.example.com/api/v4/version",
           "hostType": "gitlab",
           "username": "",
           "password": "",
           "method": "GET",
           "http2": false
         }
       }
DEBUG: Error authenticating with GitLab. Check that your token includes "api" permissions
       "err": {
         "name": "RequestError",
         "code": "UNABLE_TO_GET_ISSUER_CERT",
         "timings": {
           "start": 1708954623657,
           "socket": 1708954623661,
           "lookup": 1708954623665,
           "connect": 1708954623668,
           "error": 1708954623711,
           "phases": {"wait": 4, "dns": 4, "tcp": 3, "total": 54}
         },
         "message": "unable to get issuer certificate",
         "stack": "RequestError: unable to get issuer certificate\n    at ClientRequest.<anonymous> (/usr/local/renovate/node_modules/.pnpm/[email protected]/node_modules/got/dist/source/core/index.js:970:111)\n    at Object.onceWrapper (node:events:632:26)\n    at ClientRequest.emit (node:events:529:35)\n    at ClientRequest.emit (node:domain:489:12)\n    at ClientRequest.origin.emit (/usr/local/renovate/node_modules/.pnpm/@[email protected]/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)\n    at TLSSocket.socketErrorListener (node:_http_client:501:9)\n    at TLSSocket.emit (node:events:517:28)\n    at TLSSocket.emit (node:domain:489:12)\n    at emitErrorNT (node:internal/streams/destroy:151:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:116:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)\n    at TLSSocket.onConnectSecure (node:_tls_wrap:1659:34)\n    at TLSSocket.emit (node:events:517:28)\n    at TLSSocket.emit (node:domain:489:12)\n    at TLSSocket._finishInit (node:_tls_wrap:1070:8)\n    at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:856:12)",
         "options": {
           "headers": {
             "user-agent": "RenovateBot/37.203.2 (https://github.com/renovatebot/renovate)",
             "accept": "application/json",
             "authorization": "***********",
             "accept-encoding": "gzip, deflate, br"
           },
           "url": "https://git-csm.example.com/api/v4/version",
           "hostType": "gitlab",
           "username": "",
           "password": "",
           "method": "GET",
           "http2": false
         }
       }
FATAL: Authentication failure
 INFO: Renovate is exiting with a non-zero code due to the following logged errors
       "loggerErrors": [
         {
           "name": "renovate",
           "level": 60,
           "logContext": "FZyQYTq6Kxny2Ec-xTzSR",
           "msg": "Authentication failure"
         }
       ]

[viceice]

You shouldn't need all those extra env vars. we already test that custom root ca's work

https://github.com/containerbase/base/blob/7d702c98a0a9398cc5a094f528ac9bb12ae4bc65/test/latest/Dockerfile#L95

Please verify thsat your crt file is PEM (text) encoded and not DER (binary) encoded.

[Conor-Behard333]

@bl4ko did you find a solution to this?

[bl4ko]

no I used env to ignore err NODE_TLS_REJECT_UNAUTHORIZED=0

[viceice]

this isn't a renovate issue, you should ask on nodejs project for further help. it looks like nodejs don't like your self-signed certificate.
there's nothing we can do from renovate side.

[bl4ko]

Fixed by including full certificate chain instead of only root.crt or sub.crt (this is because renovate is based on node, and node wants full chain to verify self signed certificates).

[viceice]

makes sense, but I also think your server needs the full chain, so it can send it to clients. then the client only needs the root cert.
I had that issue a long time ago too (but with other software)