feat: Add crons task consumers #3106
Conversation
We now process tasks via Kafka consumers instead of celerybeat. This needs to be added to self-hosted as well
Codecov ReportAll modified and coverable lines are covered by tests ✅
✅ All tests successful. No failed tests found. Additional details and impacted files@@ Coverage Diff @@
## master #3106 +/- ##
=======================================
Coverage 99.01% 99.01%
=======================================
Files 3 3
Lines 203 203
=======================================
Hits 201 201
Misses 2 2 ☔ View full report in Codecov by Sentry. |
install/create-kafka-topics.sh
Outdated
| for topic in $NEEDED_KAFKA_TOPICS; do | ||
| if ! echo "$EXISTING_KAFKA_TOPICS" | grep -qE "(^| )$topic( |$)"; then | ||
| $dc exec kafka kafka-topics --create --topic $topic --bootstrap-server kafka:9092 | ||
| echo "" | ||
| fi | ||
| done | ||
|
|
||
| # This topic must have only a single partition for the consumer to work correctly | ||
| # https://github.com/getsentry/ops/blob/7dbc26f39c584ec924c8fef2ad5c532d6a158be3/k8s/clusters/us/_topicctl.yaml#L288-L295 | ||
| $dc exec kafka kafka-topics --create --topic monitors-clock-tick --bootstrap-server kafka:9092 --partitions 1 |
There was a problem hiding this comment.
Should we check to see if topic already exists here before trying to create?
I'm not totally sure if it's worth writing an integration test here. The code is well tested in Sentry, and it feels like we'd just be duplicating work. It feels like to avoid these kinds of problems it'd be more helpful to figure out how to share these kinds of configs with how things work in prod, or maybe have a way to signal that certain consumers should always be run in self hosted? |
What was the process for rolling this out to single tenant/prod? Ideally, maybe the way to go about this is to have features link to required consumers/infrastructure components and throw a warning or error in tests when pieces of infrastructure are not enabled for self-hosted? |
| # This topic must have only a single partition for the consumer to work correctly | ||
| # https://github.com/getsentry/ops/blob/7dbc26f39c584ec924c8fef2ad5c532d6a158be3/k8s/clusters/us/_topicctl.yaml#L288-L295 | ||
|
|
||
| if ! echo "$EXISTING_KAFKA_TOPICS" | grep -qE "(^| )monitors-clock-tick( |$)"; then | ||
| $dc exec kafka kafka-topics --create --topic monitors-clock-tick --bootstrap-server kafka:9092 --partitions 1 | ||
| fi |
There was a problem hiding this comment.
Question: What would happen if someone accidentally created more than 1 partition for this exact topic?
There was a problem hiding this comment.
It would cause inconsistencies in the ordering of tasks and likely cause weird behaviour. If you've done so I'd recommend repartitioning to have a single partition
|
Is it also time to consider a unified Sentry consumer at this point? |
There's a project in the works to move a lot of celery over to Kafka, so this is being considered |
We now process tasks via Kafka consumers instead of celerybeat. This needs to be added to self-hosted as well
We now process tasks via Kafka consumers instead of celerybeat. This needs to be added to self-hosted as well