These commit is mostly resolve the memory leak problem and optimize memory usage to handle millions of requests #81
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t that contains both HTTP headers and body as a single message
…ning to optimize it
…f concurrency of receiver
… pressure on GC which reduce a lot of CPU usage and memory usage when receive hugh amount number of request
…h is very useful to allow user to disable a config
…mmit prevent confiction when using Authorization header for both JWT and Pulsar token and may be other authorization purpose. Without this commit, using Authorization header as Pulsar token may cause memory leak because of token change per every request cause pulsar-beam to create a new pulsar client every time. With configurable Authorization, user can also use ?includeHeader to get Authorization header to validate a payload without casuing pulsar-beam memory leak
…Request Line in the payload to send to Pulsar, which is useful in case of Pulsar consumer would like to detect the url or method that webhook client is using
…ation with a reverse proxy engine in front of or using other Authorization method. This commit also add a short brief to let people know how to detect memory leak and resolve it. Further default configuration of Pulsar Beam should prevent this case from happening.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These commits optimize memory allocation to reuse buffer instead of keep allocating and release the memory on every request which push a lot of pressure on GC.
This pull request also add a notice and an config option to prevent memory leak if client sending Authorization while Pulsar broker does not activate it. And an option to keeps both Pulsar token authorization and normal HTTP Authorization without confliction.
This pull request also allow to set a limit on number of concurrency (config
WorkerPoolSize) to limit number of concurrent request to protect the brokers as same as leverage reusable memory buffer. Without this one, an accidentally peak time can cause crashing of service and uncontrolable of resource usage.This pull request is tested and using on production environment of https://doopage.com (we are using only Pulsar receiver) to handle few millions requests per day and it definitely improve the performance by using less CPU (mostly spend on GC) and of course much less Memory.