Use docker compose exec to create additional kafka topics #2904

saz · 2024-03-19T08:04:53Z

docker compose run will start a second kafka container, instead of reusing the existing one.
install/create-kafka-topics.sh is failing, as kafka-topics is trying to connect within the same container.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

BYK · 2024-03-19T11:40:23Z

I think you need a docker compose up kafka before everything so you have a container instance to exec in?

aldy505 · 2024-03-19T12:08:13Z

I think you need a docker compose up kafka before everything so you have a container instance to exec in?

@BYK There's this note:

self-hosted/install/create-kafka-topics.sh

Line 3 in 437bddd

 # NOTE: This step relies on `kafka` being available from the previous `snuba-api bootstrap` step 

But it'd be a good thing to include that, whether we're reusing $dc up -d kafka from the previous script or not.

BYK · 2024-03-19T12:14:26Z

Lol, I'm pretty sure I'm the one who left that note and looks like it is not very effective 😅 Also, this was pre Compose 2.0 time where we did not have the --wait option. I think adding the following as preamble would be a no-op if kafka is already running as the note suggests:

$dc up --no-build --no-recreate --wait kafka

BYK · 2024-03-19T12:15:22Z

Btw the previous script just does $dcr snuba which implies getting kafka up and in healthy state as it's a dependency of snuba. Quite implicit.

hubertdeng123 · 2024-03-19T21:38:47Z

@BYK Think you're right, CI will fail as kafka isn't up yet. Let's see what happens

aldy505 · 2024-03-19T22:06:32Z

CI didn't fail. Hubert is flabbergasted.

hubertdeng123 · 2024-03-19T22:24:51Z

Ok, running snuba-api will bring up zookeeper, redis, clickhouse, and kafka. It's from snuba_defaults

https://github.com/getsentry/self-hosted/blob/master/docker-compose.yml#L81

aldy505 · 2024-03-19T23:36:02Z

@saz can you add a line before executing everything else on the setup-kafka-topics as suggested by Burak here: #2904 (comment)

That should be all

saz · 2024-03-20T07:40:35Z

@aldy505 Done. I've also removed the comment

hubertdeng123 · 2024-03-20T17:09:44Z

Looks like we cannot use --wait yet since it's not available in v2.0.1

BYK · 2024-03-20T18:09:43Z

GAH :(

saz · 2024-03-22T16:25:00Z

Any other things I can do to solve this issue? Sticking to the implicit way isn't an option? I've seen a different issue upgrading docker compose, which will make it possible to solve this in the future.

azaslavsky · 2024-03-22T17:23:49Z

Any other things I can do to solve this issue? Sticking to the implicit way isn't an option? I've seen a different issue upgrading docker compose, which will make it possible to solve this in the future.

Is the issue with self-hosted, or is this some other unrelated problem that prevents a docker compose update?

saz · 2024-03-22T17:34:48Z

@azaslavsky it's about adding $dc up as suggested in #2904 (comment) with this PR, but --wait isn't supported in docker compose 2.0.1

aldy505 · 2024-03-23T04:16:31Z

We could just remove the --wait, and put up a script after that $dc up command to periodically check whether it's healthy yet.

while [ true ]
do
    $kafka_healthy=$(docker compose ps kafka | grep 'healthy')
    if [ ! -z "$kafka_healthy" ]; then
      break
    fi

    echo "Kafka container is not healthy, waiting for 30 seconds. If this took too long, abort the installation process, and check your Kafka configuration"
    sleep 30s
done

One alternative would be using the wait-for-it script.

saz · 2024-03-26T18:30:22Z

@aldy505 Changed as requested

install/create-kafka-topics.sh

saz · 2024-03-27T08:23:24Z

@aldy505 Changed. Thanks for spotting it!

aldy505 · 2024-03-28T13:09:34Z

Gentle ping @hubertdeng123 @BYK

BYK

LGTM, thanks! :)

BYK · 2024-03-28T13:24:55Z

install/create-kafka-topics.sh

+while [ true ]
+do
+ $kafka_healthy=$($dc ps kafka | grep 'healthy')
+ if [ ! -z "$kafka_healthy" ]; then
+ break
+ fi
+
+ echo "Kafka container is not healthy, waiting for 30 seconds. If this took too long, abort the installation process, and check your Kafka configuration"
+ sleep 30s
+done


Protip, you can do something like this:

self-hosted/_integration-test/run.sh

Line 47 in 7143d88

timeout 90 bash -c 'until $(curl -Isf -o /dev/null $SENTRY_TEST_HOST); do printf '.'; sleep 0.5; done'

Suggested change

while [ true ]

do

$kafka_healthy=$($dc ps kafka | grep 'healthy')

if [ ! -z "$kafka_healthy" ]; then

break

fi

echo "Kafka container is not healthy, waiting for 30 seconds. If this took too long, abort the installation process, and check your Kafka configuration"

sleep 30s

done

echo "Waiting for kafka container to be healthy for up to 30 seconds."

timeout 30 bash -c 'until $($dc ps kafka | grep -q 'healthy'); do printf '.'; sleep 0.5; done'

But sometimes Kafka took an entire 5 minutes to boot up :(

Well, then let's make it 300s? :D

So... Applying the change with at least 300s? Maybe it's better to even raise it a bit further, just in case?

To be honest, I'm not 100% sure. I don't use Kafka anymore on my Sentry instance, but for a minimum 4 core CPU, Kafka is rather problematic to boot up. I think it's okay to stay as it is (rather than changing to Burak's recommendation).

@BYK What's your opinionen? Would be nice to get this merged

I'm fine with whatever to be honest. I think 300s is not too bad. Maybe settle at 150? 😅

Okay let's settle this at 150s 😆

I think the reason CI is failing is because you may need to remove the $ in front of $kafka_healthy

@hubertdeng123 can you just directly commit it?

webliving · 2024-03-30T06:45:32Z

Self-Hosted Version
24.3.0

CPU Architecture
x86_64

Docker Version
26.0.0

Docker Compose Version
1.27.1

Centos Version
7.9

.install.sh error

dependency failed to start: container sentry-self-hosted-kafka-1 is unhealthy
Error in install/bootstrap-snuba.sh:3.
'$dcr snuba-api bootstrap --no-migrate --force' exited with status 1
-> ./install.sh:main:32
--> install/bootstrap-snuba.sh:source:3

Cleaning up...

aldy505 · 2024-04-14T07:34:37Z

Hey @azaslavsky @hubertdeng123 can we merge this before tomorrow's release?

azaslavsky · 2024-04-15T20:24:48Z

Running the tests now, if it passes, I'll make sure it lands before 24.4.0

azaslavsky · 2024-04-15T21:44:53Z

It looks like the tests are failing - can y'all see the action output? Ex: https://github.com/getsentry/self-hosted/actions/runs/8695624967/job/23847053353?pr=2904

saz · 2024-04-16T09:07:11Z

@azaslavsky Please approve workflows again

azaslavsky · 2024-04-16T16:59:17Z

Looks like it is still failing :(

saz · 2024-04-17T12:29:53Z

@azaslavsky I don't understand, why it's not able to attach to the container. I'm not able to reproduce this on my system :(

saz · 2024-04-17T14:51:19Z

I've found the issue. Would be great, if someone can approve the workflows again

azaslavsky · 2024-04-17T15:41:41Z

🤞

* feat: provide csrf settings information for sentry config (getsentry#2762) * feat: provide csrf settings information for sentry config * chore: trim trailing whitespace * build(deps): bump actions/setup-python from 4 to 5 (getsentry#2644) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v4...v5) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hubert Deng <[email protected]> * Fix groupedmessage indexing error (getsentry#2777) * fix groupedmessage indexing error * Check memcached backend in Django (getsentry#2778) Bail if using old memcached backend * release: 24.1.2 * build: Set master version to nightly #skip-changelog * fix: DB migration script (getsentry#2779) * use different approach to wait for postgres server * Tweak postgres indexing fix (getsentry#2792) * tweak postgres indexing fix * add exists constraint * build(deps): bump pre-commit/action from 3.0.0 to 3.0.1 (getsentry#2788) Bumps [pre-commit/action](https://github.com/pre-commit/action) from 3.0.0 to 3.0.1. - [Release notes](https://github.com/pre-commit/action/releases) - [Commits](pre-commit/action@v3.0.0...v3.0.1) --- updated-dependencies: - dependency-name: pre-commit/action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump nginx version (getsentry#2797) bump nginx version * release: 24.2.0 * build: Set master version to nightly #skip-changelog * use rust consumers in self-hosted * revert changes in 3067683 * chore: Use django ORM to perform sql commands (getsentry#2827) * use django orm * chore: provide clearer csrf url example (getsentry#2833) * Upgrade to FSL-1.1 (getsentry#2835) * simplify if for open-ai-suggestion (getsentry#2732) * simplify if for open-ai-suggestion * Add snuba rust consumers (getsentry#2831) * add snuba rust consumers * Enable proxy buffering in nginx (getsentry#2844) Enable proxy buffering Setting the proxy_buffering directive to “off” is a common mistake because it can cause performance issues and unexpected behavior in NGINX. When proxy buffering is disabled, NGINX receives a response from the proxied server and immediately sends it to the client without storing it in a buffer. This can cause problems if the response is large or if the connection between NGINX and the client is slow, because it can result in increased memory usage and potentially lead to request timeouts. To avoid this mistake, it is recommended to always enable proxy buffering in NGINX. * deps: bump maxmind/geoipupdate to 6.1.0 (getsentry#2859) https://github.com/maxmind/geoipupdate/releases/tag/v6.1.0 * increase postgres max_connections above 100 connections (getsentry#2740) * use default value and env for postgres max_connections * Integration test improvements (getsentry#2858) * integration test improvements * feat(spans): Ingest spans (getsentry#2861) * release: 24.3.0 * build: Set master version to nightly #skip-changelog * Remove duplicate feature flags (getsentry#2899) * feat: run outcomes-billing consumer (getsentry#2909) * Integration tests in python (getsentry#2892) * integration tests in python * Fix defunct java processes (getsentry#2914) revert kafka healthcheck change * Port backup tests to python (getsentry#2907) * port backup tests to python * feat(clickhouse): Added max_suspicious_broken_parts to the config.xml (getsentry#2853) * feat(clickhouse): Added max_suspicious_broken_parts to the config.xml * refactor(clickhouse): Set default max_suspicious_bronken_parts and Issue reference --------- Co-authored-by: Hubert Deng <[email protected]> * Write Customization tests in python (getsentry#2918) * port everything integration test related to python * Bump ubuntu version for tests (getsentry#2923) * bump ubuntu version used for testing * get rid of codecov cli dependency * fix(spans): Adds organizations:standalone-span-ingestion flag to default config (getsentry#2936) Adds organizations:standalone-span-ingestion flag to default config * feat: adds group attributes consumer (getsentry#2927) adds group attributes consumer * Use python for e2e tests (getsentry#2953) * bump e2e action commit sha * release: 24.4.0 * build: Set master version to nightly #skip-changelog * Port last integration tests to python (getsentry#2966) * port custom ca cert test to python * Add example to docker compose version in problem report (getsentry#2959) * Use docker compose exec to create additional kafka topics (getsentry#2904) * chore(deps): bump memcached and redis to latest patch versions (getsentry#2973) * release: 24.4.1 * build: Set master version to nightly #skip-changelog * Add workstation configuration (getsentry#2968) * Add workstation configuration These are prebuilt docker images for spinning up a local self-hosted image on the Google Cloud Workstation project. While primarily intended for internal development at Sentry, in theory these can be used by anyone with GCWS project to create a fresh workstation for developing self-hosted via a remote VSCode connection. Users who have GCWS properly configured will be able to use the forthcoming `workstations ...` command in the `sentry` dev CLI to create, manage, and destroy one-off or long-lived workstations in either the pre-install or post-install configuration. Note that the `sentry workstations ...` CLI has not yet landed in the `sentry` repo - those changes are coming soon! Issue: getsentry/team-ospo#240 * Fix shfmt complaints * Upgrade postgres to 14.11 (getsentry#2975) chore(deps): bump postgres to latest 14 alpine version * Bump docker compose version in CI (getsentry#2980) * only rerun tests on v2.0.1 * change from http error to request error * use 3 retries like before * fix: backport changes --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Reinaldy Rafli <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hubert Deng <[email protected]> Co-authored-by: Chad Whitacre <[email protected]> Co-authored-by: getsentry-bot <[email protected]> Co-authored-by: getsentry-bot <[email protected]> Co-authored-by: hubertdeng123 <[email protected]> Co-authored-by: Iven Schlenther <[email protected]> Co-authored-by: RexTim <[email protected]> Co-authored-by: Victor <[email protected]> Co-authored-by: Erfan <[email protected]> Co-authored-by: Pierre Massat <[email protected]> Co-authored-by: Jann Kleen <[email protected]> Co-authored-by: Lyn Nagara <[email protected]> Co-authored-by: edwardgou-sentry <[email protected]> Co-authored-by: Stephen Cefali <[email protected]> Co-authored-by: Edgar Sanchez <[email protected]> Co-authored-by: Steffen Zieger <[email protected]> Co-authored-by: Matthew T <[email protected]> Co-authored-by: Alex Zaslavsky <[email protected]>

…2904)

saz mentioned this pull request Mar 19, 2024

"Creating additional Kafka topics" step is failing #2903

Closed

aldy505 approved these changes Mar 19, 2024

View reviewed changes

aldy505 requested review from azaslavsky and hubertdeng123 March 19, 2024 08:07

hubertdeng123 approved these changes Mar 19, 2024

View reviewed changes

saz force-pushed the patch-1 branch from 88fc89b to dfe13e8 Compare March 20, 2024 07:39

saz force-pushed the patch-1 branch 3 times, most recently from e4adaba to 744609c Compare March 26, 2024 18:29

aldy505 reviewed Mar 27, 2024

View reviewed changes

install/create-kafka-topics.sh Outdated Show resolved Hide resolved

saz force-pushed the patch-1 branch 2 times, most recently from 0973104 to ccec185 Compare March 27, 2024 08:21

BYK approved these changes Mar 28, 2024

View reviewed changes

BYK reviewed Mar 28, 2024

View reviewed changes

saz force-pushed the patch-1 branch from 67dd698 to ef9834a Compare April 16, 2024 09:06

saz closed this Apr 16, 2024

saz force-pushed the patch-1 branch from 5b5d603 to 437bddd Compare April 16, 2024 15:48

saz reopened this Apr 16, 2024

Use docker compose exec to create additional kafka topics

09388b2

saz force-pushed the patch-1 branch from d5ce271 to 09388b2 Compare April 17, 2024 14:44

Merge branch 'master' into patch-1

6502907

azaslavsky merged commit d586cff into getsentry:master Apr 17, 2024
8 checks passed

ngudbhav pushed a commit to KingsGambitLab/self-hosted-sentry that referenced this pull request Apr 20, 2024

Use docker compose exec to create additional kafka topics (getsentry#…

85ac0f6

…2904)

ngudbhav pushed a commit to KingsGambitLab/self-hosted-sentry that referenced this pull request Apr 20, 2024

Use docker compose exec to create additional kafka topics (getsentry#…

e80e841

…2904)

github-actions bot locked and limited conversation to collaborators May 3, 2024

Use docker compose exec to create additional kafka topics #2904

Use docker compose exec to create additional kafka topics #2904

Conversation

saz commented Mar 19, 2024

Legal Boilerplate

BYK commented Mar 19, 2024

aldy505 commented Mar 19, 2024

BYK commented Mar 19, 2024

BYK commented Mar 19, 2024

hubertdeng123 commented Mar 19, 2024

aldy505 commented Mar 19, 2024 • edited

hubertdeng123 commented Mar 19, 2024 • edited

aldy505 commented Mar 19, 2024

saz commented Mar 20, 2024

hubertdeng123 commented Mar 20, 2024

BYK commented Mar 20, 2024

saz commented Mar 22, 2024

azaslavsky commented Mar 22, 2024

saz commented Mar 22, 2024

aldy505 commented Mar 23, 2024

saz commented Mar 26, 2024

saz commented Mar 27, 2024

aldy505 commented Mar 28, 2024

BYK left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webliving commented Mar 30, 2024

aldy505 commented Apr 14, 2024

azaslavsky commented Apr 15, 2024

azaslavsky commented Apr 15, 2024

saz commented Apr 16, 2024

azaslavsky commented Apr 16, 2024

saz commented Apr 17, 2024

saz commented Apr 17, 2024

azaslavsky commented Apr 17, 2024

aldy505 commented Mar 19, 2024 •

edited

hubertdeng123 commented Mar 19, 2024 •

edited