Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add a Processor Pool Service in logisland core #589

Open
oalam opened this issue Aug 31, 2021 · 0 comments
Open

add a Processor Pool Service in logisland core #589

oalam opened this issue Aug 31, 2021 · 0 comments
Labels
core framework, processor, doc, build, ... feature
Projects

Comments

@oalam
Copy link
Member

oalam commented Aug 31, 2021

pipeline objects (aka processors) are intanciated at each micro-batch by spark executors. This leads to too many objects creation and GC purge.
we could instead lazily create a pool of processors into a CoreControllerService that handles a pool of objects needed for record processing.

may be with https://commons.apache.org/proper/commons-pool/

@oalam oalam added feature core framework, processor, doc, build, ... labels Aug 31, 2021
@oalam oalam added this to To Do in Core via automation Aug 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core framework, processor, doc, build, ... feature
Projects
Core
  
To Do
Development

No branches or pull requests

1 participant