[Support]: FFMPEG memory leak with restreaming #11676
Replies: 1 comment 5 replies
-
|
This is an issue that has been reported by proxmox users before. It was sometimes found to be an issue with the GPU driver used and other times found to be a problem with the kernel used. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the quick response. Any known solutions or things to try? Also wondering if I have the setup for the lower res stream correct? |
Beta Was this translation helpful? Give feedback.
-
|
To answer my own question, looks like we’re waiting for a new pve kernel release: https://git.proxmox.com/?p=pve-kernel.git;a=shortlog Will update when its out and report back whether this fixes it. |
Beta Was this translation helpful? Give feedback.
-
|
Hello - did you find a fix for this? I think I have the exact same issue. |
Beta Was this translation helpful? Give feedback.
-
|
I did not. Apparently I need to change proxmox repos: https://forum.proxmox.com/threads/memory-leak-on-6-8-4-2-3-pve-8-2.146649/post-674376 Didn't see this message til now so will try later tonight. In the meantime I have home assistant monitoring my proxmox node memory levels and restarting the node if it gets too high in memory usage |
Beta Was this translation helpful? Give feedback.
-
|
So I updated the kernel as per that post and its significantly improved my ov time. Was previously averaging between 80 and 100ms...now it's down to 20ms. I think there was something wrong with that kernel version (see my thread here) No memory leak so far but will report back if I get one. 
|
Beta Was this translation helpful? Give feedback.
-
Describe the problem you are having
Every ~12 hours memory usage starts to dramatically climb which leads to my Frigate container running out of memory, which leads to CPU usage being pinned to 100% and Frigate dies.
The strange part is, despite errors in the logs, that object detection, recording and even HomeKit viewing all work fine.
Diving further into this issue, I can see that the FFMPEG process for my Reolink E1 Pro camera is what’s taking up all the memory.
I have restreaming set up for this camera for two reasons:
I also want to know if there is a better way to achieve (2) as I’m still making two connections to the camera. It doesn’t look like I can use one go2rtc stream as an input to another.
(Before anyone asks, the camera is in the bathroom to watch our cats, and the camera is very obvious. Home Assistant uses the PTZ to point it at the wall when it detects people in the bathroom)
Very high memory usage for FFMPEG:
Proxmox screenshot:
Version
0.13.2-6476F8A
Frigate config file
Relevant log output
FFprobe output from your camera
[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"25/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":1080,"width":1920}]}},{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"25/1","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":1440,"width":2560},{"avg_frame_rate":"0/0","codec_long_name":"AAC(AdvancedAudioCoding)"}]}}]Frigate stats
{ "cameras":{ "bathroom":{ "audio_dBFS":0.0, "audio_rms":0.0, "camera_fps":1.9, "capture_pid":381, "detection_enabled":1, "detection_fps":0.0, "ffmpeg_pid":659, "pid":366, "process_fps":1.9, "skipped_fps":0.0 }, }, "cpu_usages":{ "1":{ "cmdline":"/package/admin/s6/command/s6-svscan -d4 -- /run/service", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "15":{ "cmdline":"s6-supervise s6-linux-init-shutdownd", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "17":{ "cmdline":"/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "24":{ "cmdline":"s6-supervise frigate", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "25":{ "cmdline":"s6-supervise s6rc-oneshot-runner", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "26":{ "cmdline":"s6-supervise nginx-log", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "27":{ "cmdline":"s6-supervise go2rtc", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "28":{ "cmdline":"s6-supervise go2rtc-log", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "29":{ "cmdline":"s6-supervise frigate-log", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "30":{ "cmdline":"s6-supervise go2rtc-healthcheck", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "31":{ "cmdline":"s6-supervise s6rc-fdholder", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "32":{ "cmdline":"s6-supervise nginx", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "41":{ "cmdline":"/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "42":{ "cmdline":"/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "80":{ "cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "81":{ "cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "82":{ "cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "92":{ "cmdline":"/usr/local/go2rtc/bin/go2rtc -config=/dev/shm/go2rtc.yaml", "cpu":"1.4", "cpu_average":"0", "mem":"0.2" }, "109":{ "cmdline":"python3 -u -m frigate", "cpu":"1.8", "cpu_average":"1", "mem":"1.5" }, "110":{ "cmdline":"bash ./run.user go2rtc-healthcheck", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "117":{ "cmdline":"nginx: master process nginx", "cpu":"0.0", "cpu_average":"0", "mem":"0.1" }, "144":{ "cmdline":"nginx: worker process", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "145":{ "cmdline":"nginx: worker process", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "146":{ "cmdline":"nginx: worker process", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "147":{ "cmdline":"nginx: worker process", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "148":{ "cmdline":"nginx: cache manager process", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" }, "295":{ "cmdline":"frigate.logger ", "cpu":"0.0", "cpu_average":"0", "mem":"0.6" }, "297":{ "cmdline":"frigate.recording_manager", "cpu":"2.2", "cpu_average":"5", "mem":"1.0" }, "305":{ "cmdline":"/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(61)", "cpu":"0.3", "cpu_average":"0", "mem":"0.1" }, "306":{ "cmdline":"frigate.detector.ov ", "cpu":"29.8", "cpu_average":"29", "mem":"1.7" }, "308":{ "cmdline":"frigate.output ", "cpu":"0.6", "cpu_average":"0", "mem":"0.8" }, "321":{ "cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:", "cpu":"0.0", "cpu_average":"0", "mem":"0.1" }, "366":{ "cmdline":"frigate.process:bathroom", "cpu":"0.6", "cpu_average":"0", "mem":"0.9" }, "381":{ "cmdline":"frigate.capture:bathroom", "cpu":"0.5", "cpu_average":"0", "mem":"0.9" }, "659":{ "cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -user_agent FFmpeg Frigate/0.13.2-6476f8a -rtsp_transport tcp -timeout 5000000 -i rtsp:https://192.168.86.18:8554/bathroom -r 5 -vf fps=5,scale_vaapi=w=1280:h=720:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:", "cpu":"1.2", "cpu_average":"1", "mem":"0.4" }, "665":{ "cmdline":"ffmpeg -hide_banner -v error -allowed_media_types video -fflags nobuffer -flags low_delay -timeout 5000000 -user_agent go2rtc/ffmpeg -rtsp_flags prefer_tcp -i rtsp:https://*:*@192.168.86.36:554/h264Preview_01_main -c:v libx264 -g 50 -profile:v high -level:v 4.1 -preset:v superfast -tune:v zerolatency -pix_fmt:v yuv420p -an -vf scale=1920:1080 -user_agent ffmpeg/go2rtc -rtsp_transport tcp -f rtsp rtsp:https://127.0.0.1:8554/397c353cb2a44ff7eac784760432915e", "cpu":"14.9", "cpu_average":"15", "mem":"0.7" }, "1531":{ "cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel_flags allow_profile_mismatch -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -user_agent FFmpeg Frigate/0.13.2-6476f8a -rtsp_transport tcp -timeout 5000000 -i rtsp:https://192.168.86.18:8554/bathroom_hd -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/bathroom@%Y%m%d%H%M%S%z.mp4", "cpu":"0.8", "cpu_average":"0", "mem":"0.1" }, "2081":{ "cmdline":"sleep 30s", "cpu":"0.0", "cpu_average":"0", "mem":"0.0" } }, "detection_fps":4.0, "detectors":{ "ov":{ "detection_start":1717208578.857912, "inference_speed":247.34, "pid":306 } }, "gpu_usages":{ "intel-vaapi":{ "gpu":"13.32%", "mem":"-%" } }, "processes":{ "go2rtc":{ "pid":92 }, "logger":{ "pid":295 }, "recording":{ "pid":297 } }, "service":{ "last_updated":1717208580, "latest_version":"0.13.2", "storage":{ "/dev/shm":{ "free":228.8, "mount_type":"tmpfs", "total":256.0, "used":27.2 }, "/media/frigate/clips":{ "free":2116628.1, "mount_type":"cifs", "total":7416765.1, "used":5300137.0 }, "/media/frigate/recordings":{ "free":2116628.1, "mount_type":"cifs", "total":7416765.1, "used":5300137.0 }, "/tmp/cache":{ "free":239191.9, "mount_type":"overlay", "total":256946.1, "used":4631.0 } }, "temperatures":{ }, "uptime":487, "version":"0.13.2-6476f8a" } }Operating system
Proxmox
Install method
Docker Compose
Coral version
CPU (no coral)
Network connection
Wired
Camera make and model
Reolink E1 Pro
Any other information that may be helpful
No response
Beta Was this translation helpful? Give feedback.
All reactions