-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
K3s etcd snapshot reconcile consumes excessive memory when a large number of snapshots are present #10450
Comments
Is there actually a cumulative memory leak, or is the memory required to manage the snapshots directly proportional to the number of snapshots found on disk and in S3? If there is a cumulative memory leak, this should show up as increasing memory usage over time despite a static number of etcd snapshots. |
Seems like cumulative, amount of snapshots contribute to the time that it takes the memory to run out. By disabling S3 snapshots the issue is gone and the memory usage is stable. |
What are the units on your graph? Can you show the actual memory utilization of the k3s process in bytes? How many snapshots did you have in the cluster when you saw the memory utilization growing? I'm trying to reproduce this by profiling k3s with s3 enabled, retention set to 120, and snapshots taken 1 per minute, but I'm not quite sure that I'm seeing the exact same thing as you. |
I am also curious if you've tried adding a memory limit to the k3s systemd unit. By default the k3s systemd unit does not have a memory limit on it, and without any external memory pressure, golang will not free memory back to the operating system. So you could just be seeing secondary effects of k3s requiring more memory to reconcile a large number of snapshots, and golang not freeing memory until it absolutely needs to. |
Just to share what I'm seeing: I do see k3s allocating a lot of memory while reconciling snapshots, but this memory is freed at the end of each snapshot save cycle. Note that the memory is allocated but no longer in use, which means that it is available to be freed or reused. This is NOT a leak, but I can try to see if there is some potential for enhancement here to avoid the momentary spike in memory during reconcile. |
Just from glancing at this, I suspect that just adding some pagination to the various list operations would take the memory utilization down a lot. The current code pulls a full list into memory on every pass, which will be expensive with hundreds of snapshots. The profiling also makes it clear that this is NOT a leak, and is not related to minio. So I am going to edit the issue title to better reflect the root of the problem. |
The usage is in percents and that's a node with 8Gb of memory |
Closing based on release-1.30 results here: #10559 |
Environmental Info:
K3s Version: v1.28.10+k3s1 (a4c5612)
go version go1.21.9
but also affects some earlier versions.
Node(s) CPU architecture, OS, and Version: Linux kubernetes-worker-f-1 6.1.0-22-arm64 #1 SMP Debian 6.1.94-1 (2024-06-21) aarch64 GNU/Linux
Issue exists also atleast on ubuntu 22.04 arm64.
Cluster Configuration: 3 servers with all roles in all of them
Describe the bug:
In case when there's lots of snapshots in s3, at some point k3s will consume all memory available, and then oom killer will kick in.
Normally snapshots get cleaned, but because of bug #10292 they dont.
This will only affect single node in cluster.
Steps To Reproduce:
Expected behavior:
Memory should not run out.
Actual behavior:
Memory runs out on node, causing oom killer to start killing processes, restarting k3s service will fix the issue for some time until memory runs out again.
Additional context / logs:
Memory consumption example graph attached.
![Screenshot 2024-07-04 at 20 08 10](https://private-user-images.githubusercontent.com/814124/345905024-dbbb0ffc-3d9a-4b5b-a2b0-b9cc0a914f34.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU1NTQ4MjQsIm5iZiI6MTcyNTU1NDUyNCwicGF0aCI6Ii84MTQxMjQvMzQ1OTA1MDI0LWRiYmIwZmZjLTNkOWEtNGI1Yi1hMmIwLWI5Y2MwYTkxNGYzNC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwOTA1JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDkwNVQxNjQyMDRaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04NmI4MTc0MzJmODEwZTUwMmZkYjBkNjI1MTJjN2ViZGM5MmQxNjY0YzhiNmNlNjRkNDRmMDVjMjFlOGU2YjU4JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.bM6g0p6OiQNlrMubFDpLmuAnYrG_FexPjD0feELH3Wg)
The text was updated successfully, but these errors were encountered: