Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Creating additional Kafka topics" step is failed #2754

Closed
BBKiper opened this issue Jan 31, 2024 · 18 comments · Fixed by #2904
Closed

"Creating additional Kafka topics" step is failed #2754

BBKiper opened this issue Jan 31, 2024 · 18 comments · Fixed by #2904

Comments

@BBKiper
Copy link

BBKiper commented Jan 31, 2024

Self-Hosted Version

24.1.0

CPU Architecture

x86_64

Docker Version

25.0.1

Docker Compose Version

2.18.1

Steps to Reproduce

after the disk is full launching install.sh gives an error at the step Creating additional Kafka topics
Clearing all docker caches and containers doesn't help. Kafka and zookeeper volumes were also deleted before launch

Expected Result

correctly working self-hosted sentry

Actual Result

▶ Creating additional Kafka topics ...
Error in install/create-kafka-topics.sh:5.
'EXISTING_KAFKA_TOPICS=$($dcr -T kafka kafka-topics --list --bootstrap-server kafka:9092 2>/dev/null)' exited with status 1
-> ./install.sh:main:32
--> install/create-kafka-topics.sh:source:5

Event ID

No response

exec:
docker-compose run --rm kafka kafka-topics --list --bootstrap-server kafka:9092
output:
[2024-01-31 14:06:17,120] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.25.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

no changes were made to the compose file
When running kafka in the background, no critical errors were noticed in the logs. port 9092 listens on docker network address

@aldy505
Copy link
Collaborator

aldy505 commented Jan 31, 2024

Can you try and do this:

  1. sudo docker compose down, make sure everything is stopped
  2. sudo docker compose up -d kafka, only run kafka and zookeeper container
  3. sudo docker compose ps, make sure kafka and zookeeper are both on a "healthy" state. If one of them is not healthy, (for example kafka is not healty) do sudo docker compose logs kafka --since 5m, paste the logs here. Do the same thing for zookeeper if it's also not healthy.
  4. sudo docker compose run kafka kafka-topics --create --topic ingest-attachments --bootstrap-server kafka:9092
  5. If step 2 fails, sudo docker compose logs kafka --since 2m, paste the logs here.

@BBKiper
Copy link
Author

BBKiper commented Feb 1, 2024

@aldy505
3.

sentry-self-hosted-kafka-1       confluentinc/cp-kafka:5.5.7       "/etc/confluent/dock…"   kafka               About a minute ago   Up 42 seconds (healthy)       9092/tcp
sentry-self-hosted-zookeeper-1   confluentinc/cp-zookeeper:5.5.7   "/etc/confluent/dock…"   zookeeper           About a minute ago   Up About a minute (healthy)   2181/tcp, 2888/tcp, 3888/tcp

output like:
[2024-02-01 07:57:57,893] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka/172.26.0.4:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

kafka log:

sentry-self-hosted-kafka-1  | ===> User
sentry-self-hosted-kafka-1  | uid=0(root) gid=0(root) groups=0(root)
sentry-self-hosted-kafka-1  | ===> Configuring ...
sentry-self-hosted-kafka-1  | ===> Running preflight checks ... 
sentry-self-hosted-kafka-1  | ===> Check if /var/lib/kafka/data is writable ...
sentry-self-hosted-kafka-1  | ===> Check if Zookeeper is healthy ...
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.5.8-f439ca583e70862c3068a1f2a7d4d068eec33315, built on 05/04/2020 15:53 GMT
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=957e0a387140
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_222
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Azul Systems, Inc.
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/zulu-8-amd64/jre
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/etc/confluent/docker/docker-utils.jar
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.4.0-170-generic
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=1890MB
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=27305MB
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=1930MB
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper:2181 sessionTimeout=40000 watcher=io.confluent.admin.utils.ZookeeperConnectionWatcher@65b3120a
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=
sentry-self-hosted-kafka-1  | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeeper/172.26.0.2:2181. Will not attempt to authenticate using SASL (unknown error)
sentry-self-hosted-kafka-1  | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /172.26.0.3:34760, server: zookeeper/172.26.0.2:2181
sentry-self-hosted-kafka-1  | [main-SendThread(zookeeper:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeeper/172.26.0.2:2181, sessionid = 0x10004317df20000, negotiated timeout = 40000
sentry-self-hosted-kafka-1  | [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x10004317df20000 closed
sentry-self-hosted-kafka-1  | [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x10004317df20000
sentry-self-hosted-kafka-1  | ===> Launching ... 
sentry-self-hosted-kafka-1  | ===> Launching kafka ... 
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:50,947] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:51,272] WARN The package io.confluent.support.metrics.collectors.FullCollector for collecting the full set of support metrics could not be loaded, so we are reverting to anonymous, basic metric collection. If you are a Confluent custo
mer, please refer to the Confluent Platform documentation, section Proactive Support, on how to activate full metrics collection. (io.confluent.support.metrics.KafkaSupportConfig)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:51,273] WARN The support metrics collection feature ("Metrics") of Proactive Support is disabled. (io.confluent.support.metrics.SupportedServerStartable)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:52,046] INFO Starting the log cleaner (kafka.log.LogCleaner)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:52,076] INFO [kafka-log-cleaner-thread-0]: Starting (kafka.log.LogCleaner)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:52,221] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:52,238] INFO [SocketServer brokerId=1001] Created data-plane acceptor and processors for endpoint : EndPoint(0.0.0.0,9092,ListenerName(PLAINTEXT),PLAINTEXT) (kafka.network.SocketServer)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:52,239] INFO [SocketServer brokerId=1001] Started 1 acceptor threads for data-plane (kafka.network.SocketServer)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:52,283] INFO Creating /brokers/ids/1001 (is it secure? false) (kafka.zk.KafkaZkClient)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:52,297] INFO Stat of the created znode at /brokers/ids/1001 is: 499,499,1706774152291,1706774152291,1,0,0,72062204650848257,180,0,499
sentry-self-hosted-kafka-1  |  (kafka.zk.KafkaZkClient)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:52,298] INFO Registered broker 1001 at path /brokers/ids/1001 with addresses: ArrayBuffer(EndPoint(kafka,9092,ListenerName(PLAINTEXT),PLAINTEXT)), czxid (broker epoch): 499 (kafka.zk.KafkaZkClient)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:52,423] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
sentry-self-hosted-kafka-1  | [2024-02-01 07:55:52,480] INFO [SocketServer brokerId=1001] Started data-plane processors for 1 acceptors (kafka.network.SocketServer)

zookeeper log:

sentry-self-hosted-zookeeper-1  | ===> User
sentry-self-hosted-zookeeper-1  | uid=0(root) gid=0(root) groups=0(root)
sentry-self-hosted-zookeeper-1  | ===> Configuring ...
sentry-self-hosted-zookeeper-1  | ===> Running preflight checks ... 
sentry-self-hosted-zookeeper-1  | ===> Check if /var/lib/zookeeper/data is writable ...
sentry-self-hosted-zookeeper-1  | ===> Check if /var/lib/zookeeper/log is writable ...
sentry-self-hosted-zookeeper-1  | ===> Launching ... 
sentry-self-hosted-zookeeper-1  | ===> Launching zookeeper ... 
sentry-self-hosted-zookeeper-1  | [2024-02-01 07:55:19,377] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
sentry-self-hosted-zookeeper-1  | [2024-02-01 07:55:19,454] WARN o.e.j.s.ServletContextHandler@33f88ab{/,null,STOPPED} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler)
sentry-self-hosted-zookeeper-1  | [2024-02-01 07:55:19,454] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler)
sentry-self-hosted-zookeeper-1  | [2024-02-01 07:55:19,493] WARN [email protected]@33f88ab{/,null,STARTING} has uncovered http methods for path: /* (org.eclipse.jetty.security.SecurityHandler)

@aldy505
Copy link
Collaborator

aldy505 commented Feb 1, 2024

Your kafka is pretty much healthy. But it just shows that it's healthy for 1 minute, can you wait for a few more minutes and check if that's healthy after > 5 minutes of uptime?

@BBKiper
Copy link
Author

BBKiper commented Feb 1, 2024

@aldy505

sentry-self-hosted-kafka-1       confluentinc/cp-kafka:5.5.7       "/etc/confluent/dock…"   kafka               14 minutes ago      Up 13 minutes (healthy)   9092/tcp
sentry-self-hosted-zookeeper-1   confluentinc/cp-zookeeper:5.5.7   "/etc/confluent/dock…"   zookeeper           14 minutes ago      Up 14 minutes (healthy)   2181/tcp, 2888/tcp, 3888/tcp

looks ok

I ran run kafka kafka-topics again - it crashed due to timeout. The connection never works, no matter how long the command hangs

@aldy505
Copy link
Collaborator

aldy505 commented Feb 1, 2024

@BBKiper I have no idea on what's wrong right now. But I'm so curious for that issue. Is there any other custom configuration for your Sentry deployment?

@BBKiper
Copy link
Author

BBKiper commented Feb 1, 2024

@aldy505 The fact of the matter is that I didn't change anything. only data storage time and smtp settings

@mihalich1988
Copy link

mihalich1988 commented Feb 2, 2024

Self-Hosted Version

23.12.1, 24.1.0, 24.1.1

CPU Architecture

x86_64

Docker Version

25.0.2

Docker Compose Version

2.11.1

▶ Creating additional Kafka topics ...
Error in install/create-kafka-topics.sh:5.
'EXISTING_KAFKA_TOPICS=$($dcr -T kafka kafka-topics --list --bootstrap-server kafka:9092 2> /dev/null)' exited with status 1
-> ./install.sh:main:32
--> install/create-kafka-topics.sh:source:5

Cleaning up...

I have the same error. Doing a clean install, not an update.

@PythonNikolay
Copy link

I had the same problem on Ubuntu 22.04.3 LTS x64, the solution was to install on CentOS Stream release 9 x64

@hubertdeng123
Copy link
Member

What does your RAM/CPU usage look like?

@PythonNikolay
Copy link

What does your RAM/CPU usage look like?

I use 8cpu\16ram

@aldy505
Copy link
Collaborator

aldy505 commented Feb 5, 2024

Okay so, for some reasons we encountered an unknown error while creating topics. I can't reproduce this because I'm running Sentry with Redpanda instead of Kafka on my instance.

If you're upgrading from previous Sentry version, just comment out these lines:

for topic in $NEEDED_KAFKA_TOPICS; do
if ! echo "$EXISTING_KAFKA_TOPICS" | grep -qE "(^| )$topic( |$)"; then
$dcr kafka kafka-topics --create --topic $topic --bootstrap-server kafka:9092
echo ""
fi
done

Then do ./install.sh as normal. It'd skip the topics creation. It should be fine.


Side note: if anyone's interested on moving to Redpanda instead of Kafka, see my guide here on Sentry's Discord https://discord.com/channels/621778831602221064/796028405833007104/1201076383426809948

@BBKiper
Copy link
Author

BBKiper commented Feb 5, 2024

@aldy505 I solved the situation in the same way.
After launch, you can create topics manually if the installation is unsuccessful. Not very convenient, of course, but I don’t see any other option yet

@aldy505
Copy link
Collaborator

aldy505 commented Feb 5, 2024

@aldy505 I solved the situation in the same way. After launch, you can create topics manually if the installation is unsuccessful. Not very convenient, of course, but I don’t see any other option yet

Great to hear that. I'll close the issue for now. If anyone encountered the same issue, maybe I can reopen this or just create a new one :)

@Christophe31
Copy link

Christophe31 commented Feb 8, 2024

Hello @aldy505 thx for your suggestions.
The line causing the issue is not in your suggested changes (5)

If you're upgrading from previous Sentry version, just comment out these lines:

for topic in $NEEDED_KAFKA_TOPICS; do
if ! echo "$EXISTING_KAFKA_TOPICS" | grep -qE "(^| )$topic( |$)"; then
$dcr kafka kafka-topics --create --topic $topic --bootstrap-server kafka:9092
echo ""
fi
done

I tryed your redpanda workaround, it got me to a new issue,

ingest-profiles-1  |   File "/usr/local/lib/python3.11/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll
ingest-profiles-1  |     raise ConsumerError(str(error))
ingest-profiles-1  | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: profiles: Broker: Unknown topic or partition"}

@aldy505
Copy link
Collaborator

aldy505 commented Feb 8, 2024

Hello @aldy505 thx for your suggestions.

The line causing the issue is not in your suggested changes (5)

If you're upgrading from previous Sentry version, just comment out these lines:

for topic in $NEEDED_KAFKA_TOPICS; do
if ! echo "$EXISTING_KAFKA_TOPICS" | grep -qE "(^| )$topic( |$)"; then
$dcr kafka kafka-topics --create --topic $topic --bootstrap-server kafka:9092
echo ""
fi
done

I tryed your redpanda workaround, it got me to a new issue,


ingest-profiles-1  |   File "/usr/local/lib/python3.11/site-packages/arroyo/backends/kafka/consumer.py", line 416, in poll

ingest-profiles-1  |     raise ConsumerError(str(error))

ingest-profiles-1  | arroyo.errors.ConsumerError: KafkaError{code=UNKNOWN_TOPIC_OR_PART,val=3,str="Subscribed topic not available: profiles: Broker: Unknown topic or partition"}

@Christophe31 Hi, yeah you must create those topics manually. Otherwise your containers will keep restarting.

sudo docker compose exec kafka sh
(then inside the container)
rpk topic create profiles (and space separated topic names for any other topics)

@Christophe31
Copy link

I used your initializing script overriding create-kafka-topics.sh, it didn't worked at first but after a few relaunch I got it to work again, thx

@AlxGls
Copy link

AlxGls commented Feb 17, 2024

Can you try and do this:

  1. sudo docker compose down, make sure everything is stopped
  2. sudo docker compose up -d kafka, only run kafka and zookeeper container
  3. sudo docker compose ps, make sure kafka and zookeeper are both on a "healthy" state. If one of them is not healthy, (for example kafka is not healty) do sudo docker compose logs kafka --since 5m, paste the logs here. Do the same thing for zookeeper if it's also not healthy.
  4. sudo docker compose run kafka kafka-topics --create --topic ingest-attachments --bootstrap-server kafka:9092
  5. If step 2 fails, sudo docker compose logs kafka --since 2m, paste the logs here.

After upgrade from 22.x.x (didn't remember version) to 24.2.0 through 23.6.2 (as pointed in upgrade guide) I stacked with same error.
Following your guide I have this message:

~/self-hosted$ docker compose run kafka kafka-topics --create --topic ingest-attachments --bootstrap-server kafka:9092
[+] Creating 1/0
 ✔ Container sentry-self-hosted-zookeeper-1  Running                                                                                                                                                                                                              0.0s
Error while executing topic command : Topic ingest-attachments already exists
[2024-02-17 13:48:05,411] ERROR java.lang.IllegalArgumentException: Topic ingest-attachments already exists
	at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:247)
	at kafka.admin.TopicCommand$TopicService.createTopic(TopicCommand.scala:196)
	at kafka.admin.TopicCommand$TopicService.createTopic$(TopicCommand.scala:191)
	at kafka.admin.TopicCommand$AdminClientTopicService.createTopic(TopicCommand.scala:219)
	at kafka.admin.TopicCommand$.main(TopicCommand.scala:62)
	at kafka.admin.TopicCommand.main(TopicCommand.scala)
 (kafka.admin.TopicCommand$)

What can I do in this case? Please!

@aldy505
Copy link
Collaborator

aldy505 commented Feb 17, 2024

@AlxGls you can temporarily empty your install/create-kafka-topic.sh file to be just this:

echo "${_group}Creating additional Kafka topics ..."
echo "${_endgroup}"

Then run the install script using sudo ./install.sh && sudo docker compose up -d. If a Kafka topic already exist, it's actually fine and you have no issues.

@github-actions github-actions bot locked and limited conversation to collaborators Mar 4, 2024
saz added a commit to saz/self-hosted that referenced this issue Mar 19, 2024
saz added a commit to saz/self-hosted that referenced this issue Mar 20, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
Archived in project
Archived in project
Development

Successfully merging a pull request may close this issue.

7 participants