Performance hit with file-based package cache

[Gabriel-Ladzaretti]

What would you like help with?

I think I found a bug

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

lastest

Please tell us more about your question or problem

A fix to the file based package cache introduced in v37.280.4:

• fix(cache): Trigger cacache to remove expired contents #28275

causes a major performance impact on self-hosted users, especially those with long living containers (CE/EE).

Using the following reproduction repository, running v37.280.3, cleaning up 232 entries took 109ms:

$ rm /tmp/renovate_logs/37.280.3.log 2>/dev/null; \
          docker run \
          -it \
          --rm \
          -e LOG_LEVEL=debug \
          -e RENOVATE_LOG_FILE="/tmp/renovate_logs/37.280.3.log" \
          -v /tmp/renovate_logs:/tmp/renovate_logs \
          -v "$(pwd)/config.js":/usr/src/app/config.js \
          ghcr.io/renovatebot/renovate:37.280.3 >/dev/null; \
          grep "file cached entries" "/tmp/renovate_logs/37.280.3.log" | jq .
{
  "name": "renovate",
  "hostname": "c1f559a927f6",
  "pid": 10,
  "level": 20,
  "logContext": "GWsudBLqJXf1JrGN-zvi7",
  "msg": "Deleted 0 of 232 file cached entries in 109ms",
  "time": "2024-06-19T11:56:01.141Z",
  "v": 0
}

running v37.280.4, cleaning up 232 cache entries took 2758ms:

$ rm /tmp/renovate_logs/37.280.4.log 2>/dev/null; \
                    docker run \
                    -it \
                    --rm \
                    -e LOG_LEVEL=debug \
                    -e RENOVATE_LOG_FILE="/tmp/renovate_logs/37.280.4.log" \
                    -v /tmp/renovate_logs:/tmp/renovate_logs \
                    -v "$(pwd)/config.js":/usr/src/app/config.js \
                    ghcr.io/renovatebot/renovate:37.280.4 >/dev/null; \
                    grep "file cached entries" "/tmp/renovate_logs/37.280.4.log" | jq .
{
  "name": "renovate",
  "hostname": "7d7440e02a44",
  "pid": 10,
  "level": 20,
  "logContext": "BWCFkhvqbkzdyhpZrJE8G",
  "msg": "Deleted 0 of 232 file cached entries in 2758ms",
  "time": "2024-06-19T12:06:47.578Z",
  "v": 0
}

running latest, cleaning up 229 entries took 2409ms:

 $ rm /tmp/renovate_logs/latest.log 2>/dev/null; \
                    docker run \
                    -it \
                    --rm \
                    -e LOG_LEVEL=debug \
                    -e RENOVATE_LOG_FILE="/tmp/renovate_logs/latest.log" \
                    -v /tmp/renovate_logs:/tmp/renovate_logs \
                    -v "$(pwd)/config.js":/usr/src/app/config.js \
                    ghcr.io/renovatebot/renovate:latest >/dev/null; \
                    grep "file cached entries" "/tmp/renovate_logs/latest.log" | jq .
{
  "name": "renovate",
  "hostname": "d84fbe7bdd7e",
  "pid": 10,
  "level": 20,
  "logContext": "YIgvqs9BXyjRHDri-6E0r",
  "msg": "Deleted 0 of 229 file cached entries in 2409ms",
  "time": "2024-06-19T12:11:54.360Z",
  "v": 0
}

This performance hit is pretty consistent for all scan. I even have an example where cleaning the file-based package cache took almost 40 seconds for 1200~ cache entries, whereas prior to the fix, the time taken was much more reasonable.

Now this issue becomes even bigger for long living containers such as Renovate CE/EE.
One workaround is to use the redis/sqlite cache.

Main question is: is this considered a bug? should we change the default package cache to sqlite based? Any other suggestions?

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks

[rarkins]

#28169 (comment) could be relevant.

If we rm.content then we shouldn't need to do a verify. It might be the verify which takes all the time

[HonkingGoose]

Here's the issue to track this problem:

• Cleaning file-based package cache is slow, research if/when we need verify step #29795

[viceice]

should we switch to the sqlite based cache by default if sqlite is available? it's much faster.

[rarkins]

Yes but the install time compilation is unfortunately a problem for some

[HonkingGoose]

I opened an issue to track this idea:

• Use sqlite-based cache by default, when available #29869