Release train for 1.0

[astubbs]

• README cleanup/update regarding maintenance
• README regarding “alpha” state
• offset documentation
• javadoc cleanup sweep
• dead code sweep
• todo sweep
• disabled test sweep
• potentially remove streaming interfaces

Issues targeted for 0.5:

In priority order - only items marked high will block the 1.0 release.

Tech debt / refactoring:

• refactor: Poll Context object for API #223
• refactor: Shard object #224
• refactor: Partition state tracking instead of search #218
• refactor: Queue unification #219

Tech debt:

• Refactor: Live tracking of offsets as they change, so we don't need to scan for them #201
• Under some conditions, a shard (by partition or key), can get starved for attention #236
• build: Update Kafka to 3.1.0 #229
• feature: Retry count available in header #197
• build: Crank up Enforcer and turn on ossindex audit #215

Enhancements High:

• Replace raw ConsumerRecord with read only context object wrapper #216

Features High:

• Add support for parallel processing of batches / bulk sets of messages #18

Features Low:

Issues targeted for NEXT:

Fixes:

• vert.x: Run user funtions in a Vertical instead of Java thread pool #205
• Bug: ConsumerOffsetCommitter goes into failure state after broker downtime #203
• Subscribing to two or more topics with KEY ordering, results in messages of the same Key never being processed #184
• NoSuchFieldException when using consumer inherited from KafkaConsumer #195
• Message with null key lead to continuous failure when using KEY ordering #318

Tech debt:

• Major refactor: Package restructure #271
• fixes: #225 Support Java 16+ #289
• major: #200 Shared nothing architecture - Partition Events #270
• Docker naming, fixes and improvements #220
• Feature Request: Unique thread names for PC instances for logging #192
• Refactor test base #290

Enhancements High:

• feature: #242 Explicit retriable exception for cleaner ERROR logging #268
• Let user function throw retry exception, instead of raw errors which pollute log #242
• Allow customization of the ThreadPoolExecutor #78
• Enhanced retry epic #65
• Slow produce with pollAndProduce with many output messages #297
• When unexpected magic bytes occur, improve the feedback in the error shown to the user #327

Features High:

Features Low:

• Feature Request: Pause/Resume PC #193
• Circuit Breaker to stop processing records if HTTP endpoint errors cross a threshold #110

Issues targeted for NEXT NEXT:

Features:

• Ability use different consume and produce types for the key and value #175
• Add support for JDK's CompletableFuture into the API as part of the Reactor.io API, add docs #170
• Health-checks #71
• Enable changing of topic subscription before or after PC has started #187
• Feature: Allow unordered processing of messages without a key in 'KEY' ordering mode #183
• Need ability to seek to the beginning of a partition #191
• Ensure all PC API's are thread safe #186
• Expose way to know if connection to Broker is lost then later regained #185
• MockConsumer does not work #176
• Add a dead letter queue (DQL) implementation #310
• Have processing functions attached to topics instead of a global one for all subscribed topics #372

Tech debt:

• Consider adopting error prone and checker #259
• Adopt Truth Generator #247
• Consider turning on code sorting in the formatter #246
• Parallel join technique documentation #81

Issues targeted for post 1.0:

Features high:

• performance: Async / non blocking producer sends #232
• Improve test suite speed to pre-batch merge levels #226
• build: Update to java 17 #225
• Try refactoring WC type from String to Enum #241

Examples Low:

• How to take a single message, and distribute it to several http end points in parallel, with DLQ for failure #178
• Spring boot example #171

Tech debt:

• Refactor OffsetMapCodecManager.java #233
• Matrix test against multiple AK and JDK versions in GH #103
• Refactor: Consider a shared nothing architecture, to reduce thread complexity #200
• ProducerManager should handle different types of transaction failures appropriately #144

Features low:

• Provide option for max retires, and a call back when reached (potential DLQ) #196

[sribee]

@astubbs - This is a very interesting and useful project for improving the kafka consumer performance and also to reduce the load on the broker. Is there a timeline you could provide for the 1.0 release? thanks in advance.

[astubbs]

Yes, will be Q1 22. Once I finish up some work on another project that I'm on, I'm coming back to PC to finish up some things and do the release.

It should be noted, the only changes for 1.0 will be a couple new features, some tidy up, tighten up some of the API and things. There are no known bugs or issues that would be fixed.

[astubbs]

Set target issues in priority order. Only high priority items will block 1.0

[raphaelauv]

#18 is in high priority , that's cool !

[astubbs]

#18 is in high priority , that's cool !

Batching is now drafted complete, and awaiting merge early next week!

• Add support for parallel processing of batches / bulk sets of messages #18
• major: #18 Batch support and engine event system improvements #19

I know I know, it's been waiting a while - thanks for the help @1123 !

[astubbs]

#18 is in high priority , that's cool !

Batching is now drafted complete, and awaiting merge early next week!

Merged!

[astubbs]

FYI 0.5 is out

[colinhunt]

Hello, just inquiring if we can get a progress update on this. I am investigating using this at my company, but it still being on version 0 is a potential hindrance for adoption. Thanks!

[raphaelauv]

scikit-learn have been on v0 for more than 12 years 😁