Impossible to upgrade to latest release 24.5.0

[gabsoftware]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)
Docker in /data/docker instead of /var/lib/docker

Steps to Reproduce

Install 24.4.2 release
cd self-hosted
git fetch
git checkout 24.5.0
./install.sh

It gave me :

dependency failed to start: container sentry-self-hosted-zookeeper-1 is unhealthy
Error in install/bootstrap-snuba.sh:3.
'$dcr snuba-api bootstrap --no-migrate --force' exited with status 1
-> ./install.sh:main:36
--> install/bootstrap-snuba.sh:source:3

Expected Result

No error during upgrade

Actual Result

Error during upgrade

dependency failed to start: container sentry-self-hosted-zookeeper-1 is unhealthy
Error in install/bootstrap-snuba.sh:3.
'$dcr snuba-api bootstrap --no-migrate --force' exited with status 1
-> ./install.sh:main:36
--> install/bootstrap-snuba.sh:source:3

Product Area

Unknown

Link

No response

DSN

No response

Version

24.5.0

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[gabsoftware]

I found the culprit.

It's the script install/update-docker-volume-permissions.sh

It contains the docker path as a hard-coded value. But my docker isn't located here! You should never assume the location of Docker and use the real location.

  zookeeper_data_dir="/var/lib/docker/volumes/sentry-zookeeper/_data"
  kafka_data_dir="/var/lib/docker/volumes/sentry-kafka/_data"
  zookeeper_log_data_dir="/var/lib/docker/volumes/${COMPOSE_PROJECT_NAME}_sentry-zookeeper-log/_data"

I made a symlink of my Docker path to /var/lib/docker and it ran successfully after.

[gabsoftware]

self-hosted/install/update-docker-volume-permissions.sh

Lines 6 to 9 in f22bc2a

	zookeeper_data_dir="/var/lib/docker/volumes/sentry-zookeeper/_data"
	kafka_data_dir="/var/lib/docker/volumes/sentry-kafka/_data"
	zookeeper_log_data_dir="/var/lib/docker/volumes/${COMPOSE_PROJECT_NAME}_sentry-zookeeper-log/_data"
	chmod -R a+w $zookeeper_data_dir $kafka_data_dir $zookeeper_log_data_dir && returncode=$? || returncode=$?

[gabsoftware]

I saw that this is corrected in master.

https://github.com/getsentry/self-hosted/blob/master/install/update-docker-volume-permissions.sh

[gabsoftware]

Pity that this is not in the new release!

[TheReptile]

Another issue with that script is it assumes you're running the script as root.
If not, the chmod will fail but the install script will continue!

▶ Ensuring Kafka and Zookeeper volumes have correct permissions ...
chmod: cannot access '/var/lib/docker/volumes/sentry-zookeeper/_data': Permission denied
chmod: cannot access '/var/lib/docker/volumes/sentry-kafka/_data': Permission denied
chmod: cannot access '/var/lib/docker/volumes/sentry-self-hosted_sentry-zookeeper-log/_data': Permission denied
WARNING: Error when setting appropriate permissions for zookeeper, kafka, and zookeeper log docker volumes. This may corrupt your self-hosted install. See https://github.com/confluentinc/kafka-images/issues/127 for context on why this was added.

[aldy505]

Another issue with that script is it assumes you're running the script as root. If not, the chmod will fail but the install script will continue!

▶ Ensuring Kafka and Zookeeper volumes have correct permissions ...
chmod: cannot access '/var/lib/docker/volumes/sentry-zookeeper/_data': Permission denied
chmod: cannot access '/var/lib/docker/volumes/sentry-kafka/_data': Permission denied
chmod: cannot access '/var/lib/docker/volumes/sentry-self-hosted_sentry-zookeeper-log/_data': Permission denied
WARNING: Error when setting appropriate permissions for zookeeper, kafka, and zookeeper log docker volumes. This may corrupt your self-hosted install. See https://github.com/confluentinc/kafka-images/issues/127 for context on why this was added.

Yes, this is also being addressed here #3069

[farfromrefug]

@aldy505 @hubertdeng123 I tried #3084 locally and i still fail to install with

dependency failed to start: container for service "zookeeper" is unhealthy
Error in install/bootstrap-snuba.sh:3.
'$dcr snuba-api bootstrap --no-migrate --force' exited with status 1
-> ./install.sh:main:38
--> install/bootstrap-snuba.sh:source:3

Looking at zookeeper logs i see

Command [/usr/local/bin/dub path /var/lib/zookeeper/data writable] FAILED !
2024/05/30 14:15:30	stdout	===> Check if /var/lib/zookeeper/data is writable ...
2024/05/30 14:15:30	stdout	===> Running preflight checks ... 
2024/05/30 14:15:23	stdout	===> Configuring ...
2024/05/30 14:15:23	stdout	uid=1000(appuser) gid=1000(appuser) groups=1000(appuser) 

I ran the command sudo docker run --rm -v "sentry-zookeeper:/sentry-zookeeper-data" busybox ls -la /sentry-zookeeper-data and it gives me:

drwxrwxrwx    2 root     root          4096 May 30 11:50 .
drwxr-xr-x   20 root     root          4096 May 30 12:22 ..

Any idea? (seems the permissions should be ok)

EDIT: realized there was also an error in clickhouse:

Poco::Exception. Code: 1000, e.code() = 0, Exception: Failed to merge config with '/etc/clickhouse-server/config.d/sentry.xml': Access to file denied: /etc/clickhouse-server/config.d/sentry.xml, Stack trace (when copying this message, always include the lines below): 

not sure if it is related

[hubertdeng123]

@farfromrefug If you've checked the permissions of the volume and it seems ok, I'm wondering if you're mounting the volume correctly to /var/lib/zookeeper/data. The clickhouse issue seems interesting, maybe the config file there also needs to have permissions edited. Although that is not something I have seen before.

This is the permissions that I have

drwxrwxrwx    3 root     root          4096 May 28 20:49 .
drwxr-xr-x    1 root     root          4096 May 31 23:07 ..
drwxrwxrwx    2 root     root          4096 May 29 18:33 version-2

[farfromrefug]

@hubertdeng123 thanks à lot for taking the time to answer. You are right it is a permission issue. I just checked and it is was the local file from the repo which had permission issue. It was missing other read permission.after changing it that error is gone.
I am no facing other errors which seem not related to this issue thread (found others about it)
Thanks

[aldy505]

Since 24.5.1 is out and the fix is included, can you try upgrading directly to 24.5.1?

[aamarques]

Hey.
Same here and I didn't figure out how to solve it.

• The first issues was related to clickhouse and updating docker solve it.
• Now I couldn't fix the issue with zookeeper.

 Container sentry-self-hosted-zookeeper-1  Error
dependency failed to start: container sentry-self-hosted-zookeeper-1 is unhealthy
Error in install/bootstrap-snuba.sh:4.
'$dcr snuba-api bootstrap --no-migrate --force' exited with status 1
-> ./install.sh:main:36
--> install/bootstrap-snuba.sh:source:4

So I can't upgrade to 24.5.0

Any help ?

[aldy505]

Hey. Same here and I didn't figure out how to solve it.

• The first issues was related to clickhouse and updating docker solve it.
• Now I couldn't fix the issue with zookeeper.

 Container sentry-self-hosted-zookeeper-1  Error
dependency failed to start: container sentry-self-hosted-zookeeper-1 is unhealthy
Error in install/bootstrap-snuba.sh:4.
'$dcr snuba-api bootstrap --no-migrate --force' exited with status 1
-> ./install.sh:main:36
--> install/bootstrap-snuba.sh:source:4

So I can't upgrade to 24.5.0

Any help ?

@aamarques Usually if it's Kafka or Zookeeper issues, is that you're short on RAM or CPU cores. Perhaps try to bump your RAM or add some swap space. Personally for Sentry deployments, I recommend 1x GB of RAM + 1x GB of swap space (so if you have 16 GB of RAM, at least you should have 16 GB of swap).

[getsantry]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you remove the label Waiting for: Community, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀