major: #18 Batch support and engine event system improvements #19

astubbs · 2020-11-04T12:37:40Z

#18 - parallel batching - this PR changes the engine to always use lists of records, where non batch processing is the case where the batch size is one.

Add support for parallel processing of batches / bulk sets of messages #18
Vertx
Reactor
Docs
vertx batch test
reactor batch test
Failure tests
ensure loop cycle is good
todos
investigate build speed change - Improve test suite speed to pre-batch merge levels #226

Improve test suite speed to pre-batch merge levels #226
refactor: Poll Context object for API #223 - makes all methods batch capable and removes need for special batch methods

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelConsumer.java

...el-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelEoSStreamProcessor.java

ismarslomic · 2021-01-07T07:17:41Z

I think documentation should clarify relation between the Consumer property max.poll.records and the argument batchLevel of pollbatch(). Should we set both or only the latter? If both, should they be set to same value?

In addition it would be useful if documentation clarifies how error handling and offset commits are handled in batch processing

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelConsumer.java

astubbs · 2021-01-11T23:18:12Z

I think documentation should clarify relation between the Consumer property max.poll.records and the argument batchLevel of pollbatch(). Should we set both or only the latter? If both, should they be set to same value?

good suggestion! Will do so - hold me to it.

In addition it would be useful if documentation clarifies how error handling and offset commits are handled in batch processing

what sort of errors are you thinking about?

Off the top of my head, by default, if there is an error processing a batch of messages, the same thing would happen as if for a single message - the messages will all be returned to the internal queues, and will be retried after the default delay expires.

ismarslomic · 2021-01-12T19:42:31Z

what sort of errors are you thinking about?

Off the top of my head, by default, if there is an error processing a batch of messages, the same thing would happen as if for a single message - the messages will all be returned to the internal queues, and will be retried after the default delay expires

I was not thinking on specific errors, but in general it would be useful to have a section in docs that states exactly what you are saying above, so users are aware of their responsibilities in this aspect!

New feature request - "Retry settings"
It would be nice to have some sort of flexibility to configure the built in retry mechanisms in pollBatch(), for example enableRetry, maxRetryAttempts and waitDuration. Then you can either configure it for your needs or choose to solve it yourself.

astubbs · 2021-01-14T01:30:09Z

New feature request - "Retry settings"
It would be nice to have some sort of flexibility to configure the built in retry mechanisms in pollBatch(), for example enableRetry, maxRetryAttempts and waitDuration. Then you can either configure it for your needs or choose to solve it yourself.

Thanks again for your input! We should track this in a different feature, it's something I've been planning. I've created: #65 Enhanced retry epic. Let's continue the conversation there, and let me know if you have any further thoughts! I'll do something basic for the next version (enable/disable and then skip or die probably)...

astubbs · 2021-01-14T21:39:41Z

@ismarslomic @JorgenRingen do you have any ordering (or relative ordering) expectations or requirements for the messages within the batches?

astubbs · 2021-01-15T00:09:30Z

ok, implementation is ready for testing. I'm not entirely satisfied with the interface, but let me know what you think. I.e. we could make batching the only interface, and have the default batch size be == 1. But then the user function would always need to be a consumer which takes a list of records, which is kind of odd. Depends if you think batching would be the more common use case, or if single record processing would be..

astubbs · 2021-01-21T06:04:16Z

Does anyone have time to test this branch out? If not that's ok, I'll release it as 3.1.

...el-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelEoSStreamProcessor.java

build-parallel-consumer-core-without-tests.sh

parallel-consumer-core/pom.xml

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelConsumerOptions.java

...el-consumer-core/src/main/java/io/confluent/parallelconsumer/internal/DynamicLoadFactor.java

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/state/WorkManager.java

parallel-consumer-core/src/test/java/io/confluent/parallelconsumer/BatchTest.java

astubbs

step

astubbs

great! ;)

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelConsumerOptions.java

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/state/WorkManager.java

...el-consumer-core/src/main/java/io/confluent/parallelconsumer/internal/DynamicLoadFactor.java

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelConsumerOptions.java

parallel-consumer-core/src/test/java/io/confluent/parallelconsumer/BatchTest.java

astubbs · 2022-02-25T17:13:04Z

ok, there’s a couple test failures still - i think they need updating. But the batching feature seems feature complete now. Need to implement some test for the vertx and reactor versions, but should have this merged early next week.

astubbs

minor

...el-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelEoSStreamProcessor.java

.../src/test/java/io/confluent/parallelconsumer/AbstractParallelEoSStreamProcessorTestBase.java

parallel-consumer-core/src/test/java/io/confluent/parallelconsumer/BatchTestMethods.java

...el-consumer-reactor/src/test/java/io/confluent/parallelconsumer/reactor/CopiedBatchTest.java

...l-consumer-reactor/src/test/java/io/confluent/parallelconsumer/reactor/ReactorBatchTest.java

parallel-consumer-vertx/src/test/java/io/confluent/parallelconsumer/vertx/VertxBatchTest.java

pom.xml

astubbs

getting there... lots of improvements / refactoring in this to

...src/main/java/io/confluent/parallelconsumer/internal/AbstractParallelEoSStreamProcessor.java

...el-consumer-core/src/main/java/io/confluent/parallelconsumer/internal/DynamicLoadFactor.java

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/state/ShardManager.java

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/state/WorkContainer.java

...er-vertx/src/main/java/io/confluent/parallelconsumer/vertx/VertxParallelStreamProcessor.java

...l-consumer-reactor/src/main/java/io/confluent/parallelconsumer/reactor/ReactorProcessor.java

README.adoc

astubbs · 2022-03-10T22:33:30Z

Target should be 100% now. Protect for negative delta.

.../src/test/java/io/confluent/parallelconsumer/AbstractParallelEoSStreamProcessorTestBase.java

- test for observing dropping average batch size - performance: process mailbox timeout calculation not needed as poll box is interrupted either with work results, or with new messages from broker poller - improvements to event based system and LongPollingMockConsumer - eventing and timing improvements and simplifications / refactorings, better batch test code reuse - only sleep for retry delay if starved - remove clean/dirty state cache - too complex and unnecessary - stay tuned for replacement - change close ordering of operations to fix final commit execution - teach ProgressTracker to use time - batch size request - always request enough to fill all batches Co-authored-by: Antony Stubbs <[email protected]> Co-authored-by: Benedikt Linse <[email protected]>

astubbs self-assigned this Nov 4, 2020

astubbs mentioned this pull request Nov 4, 2020

Add support for parallel processing of batches / bulk sets of messages #18

Closed

astubbs marked this pull request as draft November 4, 2020 12:37

astubbs force-pushed the interface branch from b767af0 to a46f6f8 Compare November 4, 2020 12:46

astubbs force-pushed the features/batching branch from bb29be2 to 113c94b Compare November 4, 2020 12:46

mhowlett reviewed Nov 4, 2020

View reviewed changes

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelConsumer.java Outdated Show resolved Hide resolved

mhowlett reviewed Nov 4, 2020

View reviewed changes

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelConsumer.java Outdated Show resolved Hide resolved

mhowlett reviewed Nov 4, 2020

View reviewed changes

...el-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelEoSStreamProcessor.java Outdated Show resolved Hide resolved

Base automatically changed from interface to master November 5, 2020 11:42

astubbs force-pushed the features/batching branch from 113c94b to abe8782 Compare November 5, 2020 17:05

astubbs added the enhancement New feature or request label Nov 5, 2020

astubbs linked an issue Nov 5, 2020 that may be closed by this pull request

Add support for parallel processing of batches / bulk sets of messages #18

Closed

astubbs force-pushed the master branch 6 times, most recently from 42029e9 to 8fe9766 Compare November 10, 2020 14:32

JorgenRingen reviewed Jan 7, 2021

View reviewed changes

parallel-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelConsumer.java Outdated Show resolved Hide resolved

astubbs force-pushed the features/batching branch from abe8782 to 6d9af27 Compare January 14, 2021 22:37

JorgenRingen reviewed Jan 22, 2021

View reviewed changes

...el-consumer-core/src/main/java/io/confluent/parallelconsumer/ParallelEoSStreamProcessor.java Outdated Show resolved Hide resolved

astubbs force-pushed the master branch 2 times, most recently from bf382c4 to 97d09fc Compare March 30, 2021 08:52

astubbs commented Feb 4, 2022

View reviewed changes

astubbs force-pushed the master branch 2 times, most recently from 4c62a9f to b5b166f Compare February 16, 2022 17:41

nioertel mentioned this pull request Feb 22, 2022

Partition removal state management and concurrency improvements #190

Merged

4 tasks

astubbs force-pushed the features/batching branch from 7a40a9f to 851ce29 Compare February 25, 2022 12:28

astubbs commented Feb 25, 2022

View reviewed changes

astubbs mentioned this pull request Feb 25, 2022

Release train for 1.0 #172

Open

64 tasks

astubbs commented Feb 28, 2022

View reviewed changes

astubbs force-pushed the features/batching branch from 86ca22d to ad2e0be Compare February 28, 2022 15:34

astubbs force-pushed the master branch from 5be0137 to 84c7522 Compare March 4, 2022 13:26

astubbs changed the title ~~feature: #18 Add simple parallel batching support~~ feature: #18 Add parallel batching support Mar 4, 2022

astubbs commented Mar 4, 2022

View reviewed changes

astubbs changed the title ~~feature: #18 Add parallel batching support~~ major: #18 Batch support and engine event system improvements Mar 9, 2022

astubbs force-pushed the features/batching branch 3 times, most recently from 2fba344 to cc1167e Compare March 9, 2022 16:06

astubbs mentioned this pull request Mar 10, 2022

Unify batching API #222

Closed

astubbs commented Mar 10, 2022

View reviewed changes

README.adoc Show resolved Hide resolved

build: Upgrade TestContainer and CP

44274bc

astubbs force-pushed the features/batching branch from 3122d14 to 16d39ec Compare March 11, 2022 12:50

astubbs commented Mar 11, 2022

View reviewed changes

.../src/test/java/io/confluent/parallelconsumer/AbstractParallelEoSStreamProcessorTestBase.java Outdated Show resolved Hide resolved

astubbs force-pushed the features/batching branch 2 times, most recently from 96eb5a0 to b7b631c Compare March 14, 2022 15:15

astubbs force-pushed the features/batching branch from b7b631c to 7e26075 Compare March 14, 2022 15:26

astubbs marked this pull request as ready for review March 14, 2022 15:45

astubbs merged commit 360ebb2 into master Mar 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

major: #18 Batch support and engine event system improvements #19

major: #18 Batch support and engine event system improvements #19

astubbs commented Nov 4, 2020 •

edited

Loading

ismarslomic commented Jan 7, 2021

astubbs commented Jan 11, 2021

ismarslomic commented Jan 12, 2021

astubbs commented Jan 14, 2021

astubbs commented Jan 14, 2021

astubbs commented Jan 15, 2021

astubbs commented Jan 21, 2021

astubbs left a comment

astubbs left a comment

astubbs commented Feb 25, 2022

astubbs left a comment

astubbs left a comment

astubbs commented Mar 10, 2022

major: #18 Batch support and engine event system improvements #19

major: #18 Batch support and engine event system improvements #19

Conversation

astubbs commented Nov 4, 2020 • edited Loading

ismarslomic commented Jan 7, 2021

astubbs commented Jan 11, 2021

ismarslomic commented Jan 12, 2021

astubbs commented Jan 14, 2021

astubbs commented Jan 14, 2021

astubbs commented Jan 15, 2021

astubbs commented Jan 21, 2021

astubbs left a comment

Choose a reason for hiding this comment

astubbs left a comment

Choose a reason for hiding this comment

astubbs commented Feb 25, 2022

astubbs left a comment

Choose a reason for hiding this comment

astubbs left a comment

Choose a reason for hiding this comment

astubbs commented Mar 10, 2022

astubbs commented Nov 4, 2020 •

edited

Loading