How to limit internal queues size ?

[mikhelef]

Hello,

Is there a way to limit parallel consumer internal queues size in order to reduce RAM usage ?

Thank you.

Regards,
Massi

[rkolesnev]

Hi @mikhelef,

Configurable internal queue (buffer) size was implemented in Parallel Consumer version 0.5.2.8 - #682.
It is calculated as maxConcurrency * bufferSize * loadFactor.
#682 exposes new options in the ParallelConsumerOptions builder to allow to set load factor min/max values or set fixed value that is closest (above/ceiling) to desired queue size.

initialLoadFactor - allows to override starting / initial load factor (default 2).
maximumLoadFactor - allows to override maximum load factor that can be reached under load (default 100).

messageBufferSize - sets both initialLoadFactor and maximumLoadFactor to a fixed value closest to desired buffer size - calculated as ceiling (messageBufferSize / (maxConcurrency * bufferSize)).
messageBufferSize takes precedence over initialLoadFactor and maximumLoadFactor if both are specified.
For example with maxConcurrency of 8, batch size 1 specifying messageBufferSize as 70 will result in fixed load factor of 7 resulting in actual buffer size of 72 as load factor is a whole number and doesn't allow for fractional multiplier.

[mikhelef]

Thank you @rkolesnev !