Assign Parallel Consumer to a specific partition assignment.

[Jayesh-pict]

I have 10 partitions in my topic and am producing messages with 100 different keys i.e. 10 keys per partition. I want key-based ordering and records from different keys from a partition must be processed parallelly so I am using parallel consumer with key-based ordering. Now I want to scale my system horizontally and have multiple parallel consumers at different nodes. How can I assign one parallel consumer per partition in this system? Take into consideration that my partition count can increase based on requirements.
This worked fine when I was using the Thread per consumer model. i.e. KafkaConsumer<> with consumer.assign() method how can I achieve similar behavior with the parallel consumer?

[rkolesnev]

Kafka Consumer protocol endeavours to distribute load between consumers. Each parallel consumer instance - wraps Kafka Consumer and presents as such to cluster.
So if you have 10 partitions and 2 Parallel Consumer instances - load will be split by 5 partitions assigned to each Parallel Consumer instance.
If you launch 1 more Parallel Consumer instance - rebalance will happen and load will be split as 4,3,3 partitions per instance.
And so on up to 10 Parallel Consumer instances when each will have 1 partition assigned.

Same happens when instances are stopped - if you go from 10 Parallel Consumer instances to 5 - each will have 2 partitions assigned.
The Kafka Consumer just has to be configured with the same Group.ID so that all instances are treated as part of the same group.

Normally you shouldn't assign specific partitions to specific instances as normally you would want Kafka to manage those allocations on scale up and scale down of instances for you - for resiliency etc.