"Creating additional Kafka topics" step is failing

[saz]

Self-Hosted Version

24.3.0

CPU Architecture

x86_64

Docker Version

25.0.4

Docker Compose Version

2.19.0

Steps to Reproduce

There's another, locked issue, see #2754

Running install.sh is failing for me. It looks like, it's related to the usage of docker compose run, as this will create a new container. By using docker compose exec, it worked for me.

Expected Result

correctly working self-hosted sentry

Actual Result

Running containers:

CONTAINER ID   IMAGE                                      COMMAND                  CREATED          STATUS                                     PORTS                          NAMES
6a5218233205   confluentinc/cp-kafka:5.5.7                "kafka-topics --list…"   1 second ago     Up Less than a second (health: starting)   9092/tcp                       sentry-kafka-run-e7550f2c77d2
7cb7fab008c3   confluentinc/cp-kafka:5.5.7                "/etc/confluent/dock…"   54 seconds ago   Up 23 seconds (healthy)                    9092/tcp                       sentry-kafka-1
36c9a085aa91   redis:6.2.13-alpine                        "docker-entrypoint.s…"   54 seconds ago   Up 54 seconds (healthy)                    6379/tcp                       sentry-redis-1
75e41d676dc8   confluentinc/cp-zookeeper:5.5.7            "/etc/confluent/dock…"   54 seconds ago   Up 53 seconds (healthy)                    2181/tcp, 2888/tcp, 3888/tcp   sentry-zookeeper-1
d8866d7221f6   clickhouse-self-hosted-local               "/entrypoint.sh"         54 seconds ago   Up 53 seconds (healthy)                    8123/tcp, 9000/tcp, 9009/tcp   sentry-clickhouse-1

sentry-kafka-run-e7550f2c77d2 logs:

root@sentry1:/srv/sentry# docker logs sentry-kafka-run-e7550f2c77d2
[2024-03-19 07:33:03,626] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.6:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:33:03,729] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.6:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:33:03,829] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.6:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:33:04,131] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.6:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:33:04,632] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.6:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:33:05,535] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.6:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:33:06,738] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.6:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:33:07,740] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.6:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:33:08,943] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.6:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

The ip address shown in the log is the one of the run container, not sentry-kafka-1. It looks like, as if it's trying to connect to itself and that's why it's failing.

root@sentry1:/srv/sentry# docker compose run --rm -T kafka kafka-topics --list --bootstrap-server kafka:9092
[+] Creating 1/0
 ✔ Container sentry-zookeeper-1  Running                                                                0.0s
[2024-03-19 07:43:53,412] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:43:53,514] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:43:53,715] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:43:53,917] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:43:54,418] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:43:55,222] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:43:56,325] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:43:57,428] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:43:58,531] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:43:59,634] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:44:00,737] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2024-03-19 07:44:01,840] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.54:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

And here with docker compose exec:

root@sentry1:/srv/sentry# docker compose exec -T kafka kafka-topics --list --bootstrap-server kafka:9092
__consumer_offsets
cdc
event-replacements
event-replacements-legacy
events
events-subscription-results
generic-events
generic-metrics-distributions-subscription-results
generic-metrics-sets-subscription-results
generic-metrics-subscription-results
group-attributes
ingest-attachments
ingest-events
ingest-metrics
ingest-monitors
ingest-occurrences
ingest-performance-metrics
ingest-replay-events
ingest-replay-recordings
ingest-sessions
ingest-transactions
metrics-subscription-results
outcomes
outcomes-billing
processed-profiles
profiles
profiles-call-tree
scheduled-subscriptions-events
scheduled-subscriptions-generic-metrics-counters
scheduled-subscriptions-generic-metrics-distributions
scheduled-subscriptions-generic-metrics-sets
scheduled-subscriptions-metrics
scheduled-subscriptions-sessions
scheduled-subscriptions-transactions
sessions-subscription-results
shared-resources-usage
snuba-attribution
snuba-commit-log
snuba-dead-letter-generic-events
snuba-dead-letter-generic-metrics
snuba-dead-letter-group-attributes
snuba-dead-letter-inserts
snuba-dead-letter-metrics
snuba-dead-letter-metrics-counters
snuba-dead-letter-metrics-distributions
snuba-dead-letter-metrics-sets
snuba-dead-letter-querylog
snuba-dead-letter-replays
snuba-dead-letter-sessions
snuba-generic-events-commit-log
snuba-generic-metrics
snuba-generic-metrics-counters-commit-log
snuba-generic-metrics-distributions-commit-log
snuba-generic-metrics-sets-commit-log
snuba-metrics
snuba-metrics-commit-log
snuba-metrics-summaries
snuba-queries
snuba-replay-events
snuba-sessions-commit-log
snuba-spans
snuba-transactions-commit-log
transactions
transactions-subscription-results
root@sentry1:/srv/sentry#

Event ID

No response

[saz]

Potential fix:

diff --git a/install/create-kafka-topics.sh b/install/create-kafka-topics.sh
index 7f9022e..e5f5ce0 100644
--- a/install/create-kafka-topics.sh
+++ b/install/create-kafka-topics.sh
@@ -2,11 +2,11 @@ echo "${_group}Creating additional Kafka topics ..."

 # NOTE: This step relies on `kafka` being available from the previous `snuba-api bootstrap` step
 # XXX(BYK): We cannot use auto.create.topics as Confluence and Apache hates it now (and makes it very hard to enable)
-EXISTING_KAFKA_TOPICS=$($dcr -T kafka kafka-topics --list --bootstrap-server kafka:9092 2>/dev/null)
+EXISTING_KAFKA_TOPICS=$($dc exec -T kafka kafka-topics --list --bootstrap-server kafka:9092 2>/dev/null)
 NEEDED_KAFKA_TOPICS="ingest-attachments ingest-transactions ingest-events ingest-replay-recordings profiles ingest-occurrences ingest-metrics ingest-performance-metrics ingest-monitors"
 for topic in $NEEDED_KAFKA_TOPICS; do
   if ! echo "$EXISTING_KAFKA_TOPICS" | grep -qE "(^| )$topic( |$)"; then
-    $dcr kafka kafka-topics --create --topic $topic --bootstrap-server kafka:9092
+    $dc exec kafka kafka-topics --create --topic $topic --bootstrap-server kafka:9092
     echo ""
   fi
 done

[aldy505]

Yes, migrating to docker compose exec instead of docker compose run will help this issue.

Anyway, it'd be great if you want to submit a PR about that!

[saz]

@aldy505 Here we go: #2904