System page: MEM% considers entire host RAM, ignoring Docker mem limit

[leccelecce]

I tend to limit the maximum memory available to my docker containers, to stop a rogue container locking up the host. I'm experimenting with the memory allocated to the Frigate container. I have a VM with 12GB allocated, which runs multiple Docker containers. Frigate was assigned 1GB memory, and recently I spotted it OOM'ing. This surprised me given all the MEM% figures were < 3% on the System page. One problem is that the detector process doesn't show mem usage which I propose to change. The other is down to the use of top to get the percentages. By default, top is not aware of any Docker-enforced limit, so it calculates MEM% as if the entire host memory is available.

After a bit of testing, it seems you can do:

/opt/frigate# cat /sys/fs/cgroup/memory.max
2147483648

...to get the max memory assigned to the container in bytes. I've tested and this path is still available even if no memory limits are set in the container - it just containers "max" instead of a number of bytes.

I am proposing to send a PR to do:

1. Get total host memory via free -b
2. Get contents of /sys/fs/cgroup/memory.max
3. If file is missing, divide top memory figure by total (should match existing % figure)
4. If found and contains "max", divide top memory figure by total (should match existing % figure)
5. If found, and contains an integer, divide top process memory into total host memory

Interested in any thoughts or concerns with this approach.

[NickM-27]

The problem is I don't know if this would help with OOM, the biggest issues with memory usage is usually downloading MP4 clips which are stored in memory, but this wouldn't show up in the current system page or as sort of any process that shows in top.

If this will continue to work as is but also support your use case then sure makes sense. But you'd need to consider the amount of time this would take, we don't want the delay to be too much when trying to view the system stats.

[leccelecce]

It would make it a lot more transparent if you were in danger though. I thought my memory usage was pretty good at around 20% total because that's what the System page told me, right up until it crashed as it was really 99%.

The MP4 point is interesting though, I will monitor the container during playback and see what happens.

In terms of delays - the cat command returns instantly for me, but technically I think it could just be read once at startup as I don't think the mem limit can be changed on a running container. free has always returned instantly for me, but I will do some research as to any accounting it relies on.

[NickM-27]

The MP4 point is interesting though, I will monitor the container during playback and see what happens.

playback inside frigate does not use mp4, mp4 is only when downloading via the download button or downloading via the api

[leccelecce]

Ah gotcha - will look at that then

[leccelecce]

PR #5011 raised. Actually ended up being slightly simpler in the end.