-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhancement] add jemalloc memory tracker #51382
Merged
murphyatwork
merged 4 commits into
StarRocks:main
from
murphyatwork:murphy_jemalloc_tracker
Sep 26, 2024
Merged
[Enhancement] add jemalloc memory tracker #51382
murphyatwork
merged 4 commits into
StarRocks:main
from
murphyatwork:murphy_jemalloc_tracker
Sep 26, 2024
+97
−1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
murphyatwork
force-pushed
the
murphy_jemalloc_tracker
branch
from
September 25, 2024 07:30
47ceb37
to
42c4971
Compare
Signed-off-by: Murphy <[email protected]>
murphyatwork
force-pushed
the
murphy_jemalloc_tracker
branch
from
September 25, 2024 07:35
42c4971
to
d622558
Compare
Signed-off-by: Murphy <[email protected]>
satanson
previously approved these changes
Sep 25, 2024
trueeyu
reviewed
Sep 25, 2024
trueeyu
reviewed
Sep 25, 2024
Signed-off-by: Murphy <[email protected]>
murphyatwork
force-pushed
the
murphy_jemalloc_tracker
branch
from
September 25, 2024 09:30
d827201
to
2883dca
Compare
[Java-Extensions Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[FE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[BE Incremental Coverage Report]✅ pass : 51 / 51 (100.00%) file detail
|
murphyatwork
changed the title
[Enhancement] add jemalloc metadata tracker
[Enhancement] add jemalloc memory tracker
Sep 25, 2024
trueeyu
approved these changes
Sep 26, 2024
dirtysalt
approved these changes
Sep 26, 2024
@Mergifyio backport branch-3.3 |
@Mergifyio backport branch-3.2 |
✅ Backports have been created
|
✅ Backports have been created
|
mergify bot
pushed a commit
that referenced
this pull request
Sep 26, 2024
Signed-off-by: Murphy <[email protected]> (cherry picked from commit 2b4bced)
42 tasks
mergify bot
pushed a commit
that referenced
this pull request
Sep 26, 2024
Signed-off-by: Murphy <[email protected]> (cherry picked from commit 2b4bced)
42 tasks
wanpengfei-git
pushed a commit
that referenced
this pull request
Sep 26, 2024
Co-authored-by: Murphy <[email protected]>
wanpengfei-git
pushed a commit
that referenced
this pull request
Sep 26, 2024
Co-authored-by: Murphy <[email protected]>
renzhimin7
pushed a commit
to renzhimin7/starrocks
that referenced
this pull request
Nov 7, 2024
Signed-off-by: Murphy <[email protected]> Signed-off-by: zhiminr.ren <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
Jemalloc metadata is a footprint of jemalloc, which cannot be used by starrocks process. It should be considered into the ProcessMemTracker, otherwise the memory usage would exceed the specified
mem_limit
. For some edge cases, it can exceed 10GB, which is significant.And also, there would be some fragmentation in jemalloc, which means not all of memory can be used for allocation, we need to consider them into the MemTracker.
What I'm doing:
jemalloc_metadata
MemTracker and update according tostats.metadata
from jemallocjemalloc_fragmentation
MemTracker and set it to0.3 * (rss-allocated-metadata)
jemalloc_fragmentation_ratio
, whose default value is0.3
Function configurations:
enable_jemalloc_memory_tracker
, whether enable this feature, default truejemalloc_fragmentation_ratio
, the supposed fragmentation ratio of jemallocFixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: