Error monitoring of the self-hosted installer

[ethanhs]

In this PR we:

• Add the ability to send events to sentry (currently just an event on error/exit of non-zero status)
• Print a nice stack trace when there is an error (I have some tricks to make it even nicer, such as moving things into functions)
• Prompt the user on (and save!) their preference of reporting errors to us or not (saved e.g. for future upgrading etc)

This seems to work if I sprinkle a "false" into one of the files in install/ and the grouping seems to work correctly too.

Fixes #740

[BYK]

Nice!

[aminvakil]

This will be great! Thanks!

Some questions though:

• Can non-sentry collaborator (myself :) ) view these logs and if yes, how?
• Do we have a sample log of what will be sent?

I'd also suggest adding some uuid (some hash of IP?) which gets printed and gets sent. Then we can ask users their uuid to view their logs on our instance too. I'm not sure about privacy/policy issues this may brought, but as we're already getting users' IP, I don't think a hash of that would be a problem. This can be pasted in github in public too without a problem.

[ethanhs]

Can non-sentry collaborator (myself :) ) view these logs and if yes, how?

The sentry instance is not SaaS, so I'll talk to @chadwhitacre about if this would be possible.

Do we have a sample log of what will be sent?

We don't, but I could generate one. It is the same as the default log we store locally after installation.

I'd also suggest adding some uuid (some hash of IP?) which gets printed and gets sent.

This is a good idea. I am worried that someone could just enumerate all hashes of IPs (there aren't that many IPv4s) and so it wouldn't be secret. I'll have to think about a better source for the hash but I think this is a good idea overall.

[chadwhitacre]

Can non-sentry collaborator (myself :) ) view these logs and if yes, how?

I am checking with Legal to see what we can offer here. At the very least Sentry has a feature for sharing public links to specific events, I don't have a ton of experience with it but once we start collecting data we can experiment with this to see if it's anonymized enough. Here's a contrived example (I'm not finding mention of this in the docs):

https://self-hosted.getsentry.net/share/issue/81cdf4322c86473e8fd7662aac0f8d51/

[chadwhitacre]

A few open items. Looking good so far!

[chadwhitacre]

I hate to say it but I think we should check for whether there is a sentry-cli Docker image dynamically rather than hard-coding it. That way we can ship this and then build a Docker image for arm64 at a later date decoupled from a code change and it will "just work."

[hubertdeng123]

lol, just tried running the docker run --rm -v $basedir:/work -e SENTRY_ORG=$SENTRY_ORG -e SENTRY_PROJECT=$SENTRY_PROJECT -e SENTRY_DSN=$SENTRY_DSN getsentry/sentry-cli on my mac M1 and it is working...

Perhaps it was failing in the first place due to syntax issues and we never got around to verifying that. I don't even think we need to include separate logic for arm64

[hubertdeng123]

well, pushed up changes! could use some assistance just to confirm that this works

[chadwhitacre]

Here's what I get:

$ ./install.sh 
▶ Parsing command line ...

▶ Detecting Docker platform
Detected Docker platform is linux/arm64

▶ Initializing Docker Compose ...

▶ Setting up error handling ...
Found a .reporterrors file. What does it say? yes

▶ Checking for latest commit ... 
Error in check-latest-commit.sh:3.
'false 'Oh bother!'' exited with status 1
-> ./install.sh:main:19
--> check-latest-commit.sh:source:3

Unable to find image 'getsentry/sentry-cli:latest' locally
latest: Pulling from getsentry/sentry-cli
c7ed990a2339: Pulling fs layer
18d3cc651798: Pulling fs layer
3e6a010ae3c1: Pulling fs layer
4e81a4699a2d: Pulling fs layer
0c93296acec2: Pulling fs layer
4e81a4699a2d: Waiting
0c93296acec2: Waiting
18d3cc651798: Verifying Checksum
3e6a010ae3c1: Verifying Checksum
3e6a010ae3c1: Download complete
4e81a4699a2d: Verifying Checksum
4e81a4699a2d: Download complete
c7ed990a2339: Pull complete
18d3cc651798: Pull complete
3e6a010ae3c1: Pull complete
4e81a4699a2d: Pull complete
0c93296acec2: Verifying Checksum
0c93296acec2: Download complete
0c93296acec2: Pull complete
Digest: sha256:9c214c9b56a9292c2b1dc727e096bab185aa23bafaddb290a52377e7f5022157
Status: Downloaded newer image for getsentry/sentry-cli:latest
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Event dispatched: 13f97186-4ffa-4203-8ef6-ce5abdb7128d
Cleaning up...
$

What do you think @ethanhs? Is this okay to ship?

[ethanhs]

What do you think @ethanhs? Is this okay to ship?

I think with one or two minor changes!

Do we want separate status codes to hash together as they do now?

I think this is still an issue? But I'm not sure why it is happening tbh. I wanted to hash the traceback as a means of making the issues grouped based on traceback, but it seems that isn't happening...

[chadwhitacre]

I do see an event showing up, so it appears to work despite the warning.

[chadwhitacre]

I think this is still an issue?

I'm not going to block on this. Generally the exit code will likely be stable, it was a bit of a fluke that it varied for us.

But I'm not sure why it is happening tbh.

I think it's happening because the exit code is not included in the traceback, it only appears in $cmd_exit.

[hubertdeng123]

To add more context for the warning

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

I get that for other images we use in self-hosted since we use other linux/amd64 images, so I think this is fine

[chadwhitacre]

Docker is packaged with software that will translate or emulate x86_64 machine code into ARM64 machine code on the fly; it’s slow, but the code will run.

https://pythonspeed.com/articles/docker-build-problems-mac/#solution-2-run-x86_64-docker-images-instead

Some images do not support the ARM64 architecture. You can add --platform linux/amd64 to run (or build) an Intel image using emulation.

https://docs.docker.com/desktop/mac/apple-silicon/

That makes it sound like we have to be explicit, but we're not. 🤔

[chadwhitacre]

We set DOCKER_PLATFORM though ...

self-hosted/install/detect-platform.sh

Lines 18 to 23 in 0e1239c

	if [[ "$DOCKER_ARCH" = "x86_64" ]]; then
	export DOCKER_PLATFORM="linux/amd64"
	export CLICKHOUSE_IMAGE="yandex/clickhouse-server:20.3.9.70"
	elif [[ "$DOCKER_ARCH" = "aarch64" ]]; then
	export DOCKER_PLATFORM="linux/arm64"
	export CLICKHOUSE_IMAGE="altinity/clickhouse-server:21.6.1.6734-testing-arm"

So why does it say, "and no specific platform was requested"?

[chadwhitacre]

lolsob DOCKER_PLATFORM is unique to us, it's not a Docker envvar. 🙄

[hubertdeng123]

What's also interesting is that I don't get that warning when pulling sentry-cli:latest

Unable to find image 'getsentry/sentry-cli:latest' locally
latest: Pulling from getsentry/sentry-cli
c7ed990a2339: Pulling fs layer
18d3cc651798: Pulling fs layer
3e6a010ae3c1: Pulling fs layer
4e81a4699a2d: Pulling fs layer
0c93296acec2: Pulling fs layer
4e81a4699a2d: Waiting
0c93296acec2: Waiting
18d3cc651798: Download complete
3e6a010ae3c1: Verifying Checksum
3e6a010ae3c1: Download complete
c7ed990a2339: Verifying Checksum
c7ed990a2339: Download complete
c7ed990a2339: Pull complete
18d3cc651798: Pull complete
3e6a010ae3c1: Pull complete
4e81a4699a2d: Verifying Checksum
4e81a4699a2d: Download complete
4e81a4699a2d: Pull complete
0c93296acec2: Verifying Checksum
0c93296acec2: Download complete
0c93296acec2: Pull complete
Digest: sha256:9c214c9b56a9292c2b1dc727e096bab185aa23bafaddb290a52377e7f5022157
Status: Downloaded newer image for getsentry/sentry-cli:latest
Event dispatched: 3cf243d5-19be-44c8-8c21-c486a5c1d6f3
Cleaning up...

[ethanhs]

Ok then I think this is ready, @chadwhitacre want to give it the green check-mark?

[chadwhitacre]

Oh wow @hubertdeng123 I wonder if the delta between your env and mine is related to what we're seeing on #1671?

I think we're good here though. We use DOCKER_PLATFORM to set platform in docker-compose.yml, but that does not influence the sentry-cli image.

[ethanhs]

Oh wow @hubertdeng123 I wonder if the delta between your env and mine is related to what we're seeing on #1671?

This is what I was thinking as well.

[chadwhitacre]

Okay, pushed one more commit 5df2ab6 to fail gracefully, if the sentry-cli image doesn't work for some reason let's find out early so we don't muddy the waters with a double failure during error handling.

I'm ready to merge if yinz are!