/dev/shm fills up #11983
Answered
by
NickM-27
bdkuhman
asked this question in
Bug Report
/dev/shm fills up
#11983
-
Describe the problem you are havingAfter some time running, The logs spam with Steps to reproduceStart frigate, wait, and the ui eventually stops responding. Version0.14-beta3 Frigate config fileffmpeg:
hwaccel_args: preset-vaapi
detectors:
coral:
type: edgetpu
device: pci
record:
enabled: True
retain:
days: 3
mode: all
events:
retain:
default: 30
mode: motion
mqtt:
#creds here
go2rtc:
streams:
cam1:
# cam definition - rtsp:https://
- "ffmpeg:cam1#audio=opus"
cameras:
# Define at least one camera
cam1:
ffmpeg:
inputs:
#cam def- path: rtsp:https://
cam1-restream:
ffmpeg:
inputs:
- path: rtsp:https://localhost:8554/cam1
input_args: preset-rtsp-restreamRelevant log output2024-06-15 23:22:04.795177193 2024/06/15 16:20:07 [crit] 178#178: *136 pwritev() "/dev/shm/nginx_cache/d/68/f9939117a74ff0e3b78d2befb27a768d.0019660948" failed (28: No space left on device) while reading upstream, client: ::ffff:10.42.4.249, server: , request: "GET /api/stats HTTP/1.1", upstream: "http:https://127.0.0.1:5001/stats", host: "redacted"
2024-06-15 23:22:04.795185954 2024/06/15 16:20:12 [crit] 179#179: *148 pwritev() "/dev/shm/nginx_cache/d/68/f9939117a74ff0e3b78d2befb27a768d.0019660949" failed (28: No space left on device) while reading upstream, client: ::ffff:10.42.4.249, server: , request: "GET /api/stats HTTP/1.1", upstream: "http:https://127.0.0.1:5001/stats", host: "redacted"
2024-06-15 23:22:04.795191119 ::ffff:10.42.3.1 - - [15/Jun/2024:16:20:17 -0700] "GET / HTTP/1.1" 200 6837 "-" "kube-probe/1.30" "-"
2024-06-15 23:22:04.795196185 ::ffff:10.42.3.1 - - [15/Jun/2024:16:20:17 -0700] "GET / HTTP/1.1" 200 6837 "-" "kube-probe/1.30" "-"
s6-log: warning: unable to write to /dev/shm/logs/frigate/current: No space left on device
s6-log: warning: unable to write to /dev/shm/logs/nginx/current: No space left on device
...Operating systemDebian Install methodDocker Compose Network connectionWired Camera make and modelAmcrest IP8M-2779EW-AI Any other information that may be helpfulRunning in Kubernetes w/ helm chart. (not docker-compose), Also was happening on beta2, was hoping #11834 helped. watching /dev/shm with du and ls, previously, I saw a large number of <camname><ts>.<num> files at the root of the dir. Normally I see these pop in and out. Of course after trying to observe it for this report, it's not happening but I'll update it once I see it again. From du, it looks like /dev/shm is filling up in it's root. The streams are in 4k, so I expect to have large files here occasionally. They seem to be ~12M, but 0.13 was fine, and 0.14 works fine for a bit until something happens and it suddenly fills up. |
Beta Was this translation helpful? Give feedback.
Answered by
NickM-27
Jun 21, 2024
Replies: 1 comment 8 replies
-
|
If space is increased to 4GB for SHM then something very large is taking up this space and more info is needed to know what is going on in this case |
Beta Was this translation helpful? Give feedback.
8 replies
-
|
again, linking back to #11550 |
Beta Was this translation helpful? Give feedback.
-
|
Looking into the code, Not sure why this sometimes is fine though, or how it ends up with filling /dev/shm up |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, this is the issue, will fix |
Beta Was this translation helpful? Give feedback.
-
|
Fixed in #12093 |
Beta Was this translation helpful? Give feedback.
Answer selected by
bdkuhman
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Fixed in #12093