ParallelConsumer behavior on Error

[mehdizyane]

Hello,

I'm trying to understand the behavior of ParallelConsumer on error case, I have several questions :

1. Knowing that ParellelConsumer will run forever even on error case, there is any case where the Broker will eject the consumer from the consumer group because no commit is happening ?

2. If we want to stop consuming on error how can we achieve that ?

Thank you for your time.

[rkolesnev]

Hi,

The Broker will not eject the consumer on user function error - the underlying Kafka Consumer will be alive and heartbeating with the Broker.
The Parallel Consumer will retry the processing on exception thrown in User Function - but it will indeed retry it forever. It is up to application to add retry handling code for either dropping the message on max retries or using DLQ, or other failure handling strategies.
You can stop Parallel Consumer in the User Function by calling shutdown or indicating shutdown request to main app thread and stop it from there.
Alternatively you can monitor ParallelConsumer metrics and based on number of events failing, number of events processed successfully etc decide to shut it down.
For example - shutting down from User Function on error and some condition like max retries exhausted:

        parallelConsumer.poll(recordContexts -> {
            try {
                processRecord(recordContexts);
            } catch (Exception e) {
                if (exhaustedRetries(recordContexts, e)) {
                    CompletableFuture.runAsync(() -> parallelConsumer.closeDontDrainFirst(Duration.ofSeconds(10)));
                }
            }
        });

[mehdizyane]

Thank you for the very clear reply 👍