Upgrade to 24.5 failed

[romanstingler]

Self-Hosted Version

Sentry 24.6.0.dev0 -> please read

CPU Architecture

x86-64

Docker Version

25.0.2

Docker Compose Version

1.29.2

Steps to Reproduce

I have a
Welcome to Ubuntu 23.04 (GNU/Linux 6.2.0-1020-gcp x86_64)
VM in GCP
had 24.3 installed and wanted to upgrade to 24.5

this failed because of your docker-compose version check
I had 1.28.0 or so and it failed.
So I upgraded my version checked

roman_stingler@sentry:~$ docker-compose version
docker-compose version 1.29.2, build unknown
docker-py version: 5.0.3
CPython version: 3.11.4
OpenSSL version: OpenSSL 3.0.8 7 Feb 2023

all good, and it failed again with the same issue, detects 1.28 but 1.29 is installed.

then I overwrote the check, then failed to build
because clickhouse was unkown

so i checked out the 24.4.2 version

this build was successful
and now it shows
Sentry 24.6.0.dev0

but since 18h the vm is burning

I think I have to wait for the error only version

Expected Result

see above

Actual Result

see above

Event ID

No response

My question is, why is there even python in a high performance environment?

[guiolmar]

@romanstingler I was dealing with the same problem yesterday. I read in another issue that it is necessary to go through version 24.4.2. Try checking that version, running install and then upgrading to version 24.5.1. I hope you can solve it! 😊

[romanstingler]

@guiolmar thank you for your input.
I will try upgrading after working hours

[guiolmar]

@romanstingler Nice! I await your response to know if it has worked for you 🤗

[aldy505]

The installation failed because we.. kinda bumped the minimum version of Docker Compose, that we need to execute docker compose down <service name>. Here's the PR for it #3078.

But I see you're still using the old docker-compose wrapper script. I'd suggest you to remove it and re-install the new version of docker compose, you'll be somewhat reinstalling Docker, see the guide here https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository (what you need is at the sudo apt-get install ... docker-compose-plugin command)

[romanstingler]

I tried to upgrade, failed multiple times, because some containers were already present.
what do I know.
It worked at the end, all logged issues were gone, not a big problem.
but now it doesn't log any errors

Now I reverted to the old snapshot and the web UI didn't show anything except an error
and i tried to install again with the old version

just so frustrating.

Why is there no upgrade guide?
The comment, which is maybe a partial good luck wish,
in the release page,
which is not included in the CHANGELOG.MD
is not very helpful.

Someone should take a look at Laravel's upgrade guide
https://laravel.com/docs/11.x/upgrade
you have the dependency version changes and changes you have to do.

▶ Upgrading Clickhouse ...
Container sentry-self-hosted-clickhouse-1 Starting
Container sentry-self-hosted-clickhouse-1 Started
Waiting for clickhouse server, 30 remaining attempts...
Waiting for clickhouse server, 29 remaining attempts...
Waiting for clickhouse server, 28 remaining attempts...
Waiting for clickhouse server, 27 remaining attempts...
Waiting for clickhouse server, 26 remaining attempts...
Waiting for clickhouse server, 25 remaining attempts...
Waiting for clickhouse server, 24 remaining attempts...
Waiting for clickhouse server, 23 remaining attempts...
Waiting for clickhouse server, 22 remaining attempts...
Waiting for clickhouse server, 21 remaining attempts...
Waiting for clickhouse server, 20 remaining attempts...
Waiting for clickhouse server, 19 remaining attempts...
Waiting for clickhouse server, 18 remaining attempts...
Waiting for clickhouse server, 17 remaining attempts...
Waiting for clickhouse server, 16 remaining attempts...
Waiting for clickhouse server, 15 remaining attempts...
Waiting for clickhouse server, 14 remaining attempts...
Waiting for clickhouse server, 13 remaining attempts...
Waiting for clickhouse server, 12 remaining attempts...
Waiting for clickhouse server, 11 remaining attempts...
Waiting for clickhouse server, 10 remaining attempts...
Waiting for clickhouse server, 9 remaining attempts...
Waiting for clickhouse server, 8 remaining attempts...
Waiting for clickhouse server, 7 remaining attempts...
Waiting for clickhouse server, 6 remaining attempts...
Waiting for clickhouse server, 5 remaining attempts...
Waiting for clickhouse server, 4 remaining attempts...
Waiting for clickhouse server, 3 remaining attempts...
Waiting for clickhouse server, 2 remaining attempts...
Waiting for clickhouse server, 1 remaining attempts...
Error response from daemon: Container 518e4f8c1febf44bc1608b4671f7ec7bb397e95be375bbaccab2b1787bebf6ea is restarting, wait until the container is running
Error in install/upgrade-clickhouse.sh:21.
'version=$($dc exec clickhouse clickhouse-client -q 'SELECT version()')' exited with status 1
-> ./install.sh:main:25
--> install/upgrade-clickhouse.sh:source:21

Envelope from file sentry-envelope-bf0f3f63f8a1ef8daeb52ef3ecee361a dispatched
Cleaning up...
roman_stingler@sentry:/srv/docker/sentry/self-hosted$

I restored from an old 24.3 snapshot and will keep that for the time being until I see bugfixes
#3016 (comment)
this one is probably not resolved

[jav-12]

I have the same problem on my sentry instance. Now I'm stuck because rollback doesn't work either.

I tried to install a new sentry deployment on a newly created machine and the same problem:

# ./install.sh --no-user-prompt --no-report-self-hosted-issues
▶ Parsing command line ...
WARNING The --no-user-prompt flag is deprecated. Please use --skip-user-creation instead.

▶ Detecting Docker platform
Detected Docker platform is linux/amd64

▶ Initializing Docker Compose ...

▶ Setting up error handling ...
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 292B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/debian:bookworm-slim
#2 DONE 0.4s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [1/2] FROM docker.io/library/debian:bookworm-slim@sha256:2ccc7e39b0a6f504d252f807da1fc4b5bcd838e83e4dec3e2f57b2a4a64e7214
#4 DONE 0.0s

#5 [2/2] RUN set -x   && apt-get update   && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends jq   && apt-get clean   && rm -rf /var/lib/apt/lists/*
#5 CACHED

#6 exporting to image
#6 exporting layers done
#6 writing image sha256:50746130f9ffcc9fe90d19dbea3c38bdc1c6002d0fe5371a29ff4dd9bc0f0f75 done
#6 naming to docker.io/library/sentry-self-hosted-jq-local done
#6 DONE 0.0s

▶ Checking for latest commit ...
skipped

▶ Checking minimum requirements ...
Found Docker version 27.1.2
Found Docker Compose version 2.29.1

▶ Upgrading Clickhouse ...
 Container sentry-clickhouse-1  Starting
 Container sentry-clickhouse-1  Started
Waiting for clickhouse server, 30 remaining attempts...
Waiting for clickhouse server, 29 remaining attempts...
Waiting for clickhouse server, 28 remaining attempts...
Waiting for clickhouse server, 27 remaining attempts...
Waiting for clickhouse server, 26 remaining attempts...
Waiting for clickhouse server, 25 remaining attempts...
Waiting for clickhouse server, 24 remaining attempts...
Waiting for clickhouse server, 23 remaining attempts...
Waiting for clickhouse server, 22 remaining attempts...
Waiting for clickhouse server, 21 remaining attempts...
Waiting for clickhouse server, 20 remaining attempts...
Waiting for clickhouse server, 19 remaining attempts...
Waiting for clickhouse server, 18 remaining attempts...
Waiting for clickhouse server, 17 remaining attempts...
Waiting for clickhouse server, 16 remaining attempts...
Waiting for clickhouse server, 15 remaining attempts...
Waiting for clickhouse server, 14 remaining attempts...
Waiting for clickhouse server, 13 remaining attempts...
Waiting for clickhouse server, 12 remaining attempts...
Waiting for clickhouse server, 11 remaining attempts...
Waiting for clickhouse server, 10 remaining attempts...
Waiting for clickhouse server, 9 remaining attempts...
Waiting for clickhouse server, 8 remaining attempts...
Waiting for clickhouse server, 7 remaining attempts...
Waiting for clickhouse server, 6 remaining attempts...
Waiting for clickhouse server, 5 remaining attempts...
Waiting for clickhouse server, 4 remaining attempts...
Waiting for clickhouse server, 3 remaining attempts...
Waiting for clickhouse server, 2 remaining attempts...
Waiting for clickhouse server, 1 remaining attempts...
Error response from daemon: Container c6b2b6c429f75035ff237ac90faf3538f4969ea4d1d7e015a1d0f959eb11bdf0 is restarting, wait until the container is running
Error in install/upgrade-clickhouse.sh:21.
'version=$($dc exec clickhouse clickhouse-client -q 'SELECT version()')' exited with status 1
-> ./install.sh:main:25
--> install/upgrade-clickhouse.sh:source:21

Cleaning up...

[hubertdeng123]

I'd be curious to see what logs are going on in your clickhouse container. Please open another issue for container restarting. This looks like a different issue than the original issue however, since that one is complaining about resource usage.

[unleftie]

I had a similar error, but I didn't notice how the volume ran out of free space. After I freed up some space, the problem disappeared