[Support]: /dev/shm/logs/frigate/current: No space left on device

[psilantropy]

Describe the problem you are having

Hi all,

After upgrading from the .13 to .14 Frigate seems to fill up /shm and eventually crash.
Usually has 256MB, but granted 1024MB and it just eats it all anyway.

First time i've caught this view with 100%.

Contents of /dev/shm on container, it just all these files per camera configured.

After deploy contents look like this (at 3% of 256MB)

If I refresh this directory, it constantly changes as it presumably should. But I don't seem to catch it filling up.

Version

0.14.0-beta1

Frigate config file

mqtt:
  enabled: true
  host: x.x.x.x.200
  port: 1883
  user: mqtt-user
  password: removed

record:
  enabled: true
  events:
    retain:
      default: 120
      mode: motion

database:
  path: /config/frigate.db

detectors:
  coral1:
    type: edgetpu
    device: usb:0
    model:
  coral2:
    type: edgetpu
    device: usb:1
    model:

snapshots:
  enabled: true
  bounding_box: true
  clean_copy: true
  retain:
    default: 60
    objects:
      person: 365

birdseye:
  enabled: true
  mode: continuous
  restream: true
  width: 1920
  height: 1080

go2rtc:
  streams:
    Garage:
      - rtsp:https://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    Backyard:
      - rtsp:https://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    Laundry:
      - rtsp:https://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0
    wyzev3-01:
      - rtsp:https://user:[email protected]:554/unicast

objects:
  track:
    - person
    - dog
    - cat
    - rat
    - plane
    - airplane

telemetry:
  version_check: true

cameras:

  Laundry:
    enabled: true
    birdseye:
      order: 1
    record:
      retain:
        days: 60
        mode: all
      events:
        retain:
          default: 120
          mode: active_objects
    audio:
      enabled: true
      max_not_heard: 30
      min_volume: 200
      listen:
        - bark
        - fire_alarm
        - scream
        - speech
        - yell
        - cheering
        - bang
        - smash
        - crushing
        - clang
        - beep
        - honk
        - toot
        - skidding
        - tire_squeal
        - police_car
        - ambulance
        - fire_engine
        - siren
    ffmpeg:
      hwaccel_args: preset-nvidia-h265
      inputs:
        - path: rtsp:https://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
            - audio
        - path: rtsp:https://user:[email protected]:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
      output_args:
        record: preset-record-generic-audio-aac
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 5
    mqtt:
      timestamp: true
      crop: true
      height: 720
      bounding_box: true
      quality: 100
    objects:
      track:
        - person
    motion:
      mask:
        - 1113,664,1280,664,1280,720,1115,720
        - 1280,410,1199,392,1163,373,1193,297,1280,255

  Backyard:
    enabled: true
    birdseye:
      order: 3
    record:
      retain:
        days: 30
        mode: all
      events:
        retain:
          default: 120
          mode: active_objects
    ffmpeg:
      hwaccel_args: preset-nvidia-h265
      inputs:
        - path: rtsp:https://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
            - audio
        - path: rtsp:https://user:[email protected]:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
      output_args:
        record: preset-record-generic-audio-aac
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 1280,0,1280,253,1098,161,1118,90,1020,59,1011,132,936,83,953,0
        - 561,0,327,0,0,0,0,720,439,720,497,398,509,248,581,190
    mqtt:
      timestamp: true
      crop: true
      height: 720
      bounding_box: true
      quality: 100

  Garage:
    enabled: true
    birdseye:
      order: 4
    record:
      retain:
        days: 14
        mode: all
      events:
        retain:
          default: 120
          mode: motion
    ffmpeg:
      hwaccel_args: preset-nvidia-h265
      inputs:
        - path: rtsp:https://user:[email protected]:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
            - audio
        - path: rtsp:https://user:[email protected]:554/cam/realmonitor?channel=1&subtype=2
          roles:
            - detect
      output_args:
        record: preset-record-generic-audio-aac
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 5
    mqtt:
      timestamp: true
      crop: true
      height: 720
      bounding_box: true
      quality: 100
    motion:
      mask:
        - 0,720,375,720,378,670,0,662

  wyzev3-01:
    enabled: true
    birdseye:
      enabled: true
    record:
      retain:
        days: 120
        mode: all
      events:
        retain:
          default: 365
          mode: motion
    objects:
      track:
        - dog
        - person
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      inputs:
        - path: rtsp:https://user:[email protected]:554/unicast
          roles:
            - record
    motion:
      mask:
        - 0,0,0.468,0,0.417,0.728,0.29,1,0,1
        - 0.738,-0.001,0.366,0,0.349,0.567,0.677,0.572,1,1,1,0.419,1,0
        - 0.864,0.931,0.976,0.934,0.972,0.996,0.859,0.988

Relevant log output

Not sure if this is relevant, but it's pre- no space error.

[2024-05-27 18:52:49] frigate.record.maintainer      WARNING : Unable to keep up with recording segments in cache for Backyard. Keeping the 6 most recent segments out of 7 and discarding the rest...

2024-05-27T06:52:49.482622589Z s6-log: warning: unable to write to /dev/shm/logs/frigate/current: No space left on device
2024-05-27T06:52:49.986624991Z s6-log: warning: unable to write to /dev/shm/logs/nginx/current: No space left on device

FFprobe output from your camera

I can't get this working quickly, please let me know if required for this error.

Frigate stats

No response

Operating system

Debian

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Dahua 5 series and 1 Wyze cam v3

Any other information that may be helpful

Volumes:

/portainer/Files/AppData/Config/frigate /config
/srv/dev-disk-by-uuid-xxxxxxx/frigate/clips /media/frigate/clips
/srv/dev-disk-by-uuid-xxxxxxx/frigate/recordings /media/frigate/recordings
/srv/dev-disk-by-uuid-xxxxxxx/frigate/exports /media/frigate/exports
/srv/dev-disk-by-uuid-xxxxxxx/frigate/cache /tmp/cache
/etc/localtime /etc/localtime

[psilantropy]

I just noticed I have a significant amount of updates this afternoon on Debian. Including docker, cuda etc. Just applying all those to see what changes.
I also created a tmpfs volume and made that adjustment.

[NickM-27]

We are going to need a full copy of logs

[pipip]

I have the same issue (shm quickly raising until 100%). It seems like sth is not recognized correctly (it mentiones "c210_livin" but my camera stream is named tapo-c210_livingroom, also moov is not part of my config)

I found this inthe logs:

frigate  | 2024-06-02 00:25:57.180433505  [2024-06-02 00:25:57] frigate.util.config            INFO    : Checking if frigate config needs migration...                                                               
frigate  | 2024-06-02 00:25:57.262772032  [2024-06-02 00:25:57] frigate.util.config            INFO    : frigate config does not need migration...
frigate  | 2024-06-02 00:25:58.298947965  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f58180f7dc0] moov atom not found
frigate  | 2024-06-02 00:25:58.452640482  Process output_processor:
frigate  | 2024-06-02 00:25:58.453222049  Traceback (most recent call last):
frigate  | 2024-06-02 00:25:58.453329981    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
frigate  | 2024-06-02 00:25:58.453332175      self.run()               
frigate  | 2024-06-02 00:25:58.453388130    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run                            
frigate  | 2024-06-02 00:25:58.453390291      self._target(*self._args, **self._kwargs)                                                       
frigate  | 2024-06-02 00:25:58.453433648    File "/opt/frigate/frigate/output/output.py", line 74, in output_frames                           
frigate  | 2024-06-02 00:25:58.453435812      preview_recorders[camera] = PreviewRecorder(cam_config)
frigate  | 2024-06-02 00:25:58.453481531    File "/opt/frigate/frigate/output/preview.py", line 217, in __init__
frigate  | 2024-06-02 00:25:58.453483427      ts = float(file_time)
frigate  | 2024-06-02 00:25:58.453544771  ValueError: could not convert string to float: 'c210_livin'
frigate  | 2024-06-02 00:25:58.503128206  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f58180fcc00] moov atom not found
frigate  | 2024-06-02 00:25:58.660528491  [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7f581847f0c0] moov atom not found
frigate  | 2024-06-02 00:25:59.751218546  [INFO] Starting go2rtc healthcheck service...`

[psilantropy]

Yea this has come back a few times. Usually if I edit anything in the config and save and restart.
I just redeploy the container again to fix it.

I'll see if I have the same thing next time it occurs.

[pipip]

maybe the "-" in the camera stream name causes issues. I noticed you also have a stream with "-" in it (wyzev3-01)

/edit
I have removed the "-" from my stream / cam names. This has solved the issue for me

[psilantropy]

I did, and I have actually deleted it from my config to rule out general "wyze" issues. Just Dahua 5 series at the moment. Have not had this issue for a few days.

However, performance is pretty bad. I generally have 0.0x copy speeds in my debug logs, but viewing camera history etc is pretty average.
I have recording/exports/clips all bound to a raid5 array, config and cache to nvme. Only 3 cameras in this system.

Might test on ssd later.

[psilantropy]

I'm going to blame updates for this. Nvidia, many other related driverupdates and things appear ok now. Will reopen ifneeded. Thanks.

[nomonkeynodeal]

I just had the same problem. I performed zero updates.

[apranda]

I increased my /shm size to 4GB, which reduced Frigate crashing from hourly to ~6 times per day. Did not have any issues previously on 0.13. From another thread I saw, the root cause may be PID limits, which looks like it may be fixed in the next build.

[themoosman]

The PID issue is fixed in the dev builds. I'm running a dev build.

[rdkartono]

It happened to me .
/dev/shm become full, and frigate unstable.
/tmp/cache is not the problem.

The problem happened when I run frigate with --network=host. If not, frigate stable.
I found the problem , that frigate can not listen at port 8082. It was crashed with my Seafile which also listening at that port.
I tried disabling Seafile and Seahub, and run docker , /dev/shm is normal, /tmp/cache is normal.
To confirm "normal", i tried docker run with --ipc=host option, and monitoring storage using df -h.

I check using netstat , apparently frigate is listening on port 8082 for process frigate.output. I am guessing this is used to write recording from /dev/shm and /tmp/cache to frigate media.

I think frigate should give options on which ports it want to use.
I want my seafile service too .

[NickM-27]

it is not recommended to run frigate in host mode, if you want to change the ports then you can mount your own nginx config file

[rdkartono]

In my case, I need --network=host because I need constant fresh access to /etc/hosts file, because of Tuya camera problem I mentioned in other post.
For option --ipc=host I just use to monitor /dev/shm and /tmp/cache status. I could disable it after I confident.

However running overnight with 6 cameras, It is stable.
/dev/shm below 10 MB.
/tmp/cache below 10MB.
It is overnight status which was very low on motions.
I will check again this day.

[rdkartono]

just checked this afternoon.
/dev/shm usage only 71MB .
/tmp/cache usage only 83 MB.

System has been stable for hours.
In contrast, before I found the culprit of Port 8082, /dev/shm will be full in just minutes.

[nomonkeynodeal]

/dev/shm become full for me too. I am not running in host mode. I recreated the container and it's working for now.

Edit: It happened again. I nuked the DB and all recordings and started over.

[psilantropy]

This issue came back a few times. In the end I deleted all my recordings and database and have not had the issue since.
I do not run in host mode either.

[apranda]

Deleted all recordings and database and also have had no crashes for over a day now.

Edit: It crashed again with the Python bus error. Shm size is 4GB so the issue lies elsewhere.