Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Support]: I keep getting "unable to keep up with recording segments" and my footage skips a lot. #7297

Closed
Majestic7979 opened this issue Jul 27, 2023 · 135 comments

Comments

@Majestic7979
Copy link

Majestic7979 commented Jul 27, 2023

Describe the problem you are having

Per the title Log says "unable to keep up with recording segments". I don't remember ever getting this on 0.11. I am getting lots of skips in my recordings. The server is more powerful than the one I was using before, but the HDD is the same. The only difference is the Frigate version. What's wrong with my config? I enabled G711 audio in my camera's stream, this is the only difference, but Frigate does not accept it anyway. Even with AAC it doesn't really work and never has. I tried a lot of different ways/presets. Do I have to completely disable audio in the camera to make this work?

Version

0.12.1-367D724

Frigate config file

####################################
########## FRIGATE CONFIG ##########

mqtt:
  enabled: false

detectors:
  tensorrt:
    type: tensorrt
    device: 0


model:
  path: /trt-models/yolov7-tiny-416.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 416
  height: 416

ui:
  live_mode: jsmpeg
  timezone: Europe/London
  use_experimental: true

logger:
  default: info
  logs:
    frigate.event: info

database:
  path: /db/frigate.db

birdseye:
  enabled: True
  restream: False
  width: 1920
  height: 1080
    # 1 is the highest quality, and 31 is the lowest. Lower quality feeds utilize less CPU resources.
  quality: 15
  mode: continuous

detect:
  width: 2592
  height: 1944
  fps: 5
  enabled: true
  max_disappeared: 60

objects:
  track:
    - person

record:
  enabled: True
  retain:
    days: 10
    mode: all
  events:
    retain:
      default: 10

snapshots:
  enabled: False
  timestamp: true
  bounding_box: False
  crop: False
  retain:
    default: 5

rtmp:
  enabled: False

timestamp_style:
    # "tl" (top left), "tr" (top right), "bl" (bottom left), "br" (bottom right)
  position: "tl"
    # Optional: Format specifier conform to the Python package "datetime" (default: shown below)
    #           Additional Examples:
    #             german: "%d.%m.%Y %H:%M:%S"
  format: "%m/%d/%Y %H:%M:%S"
  color:
    red: 255
    green: 255
    blue: 255
  thickness: 2
    # None (No effect),
    # "solid" (solid background in inverse color of font)
    # "shadow" (shadow for font)
  effect: shadow

###################################
########## CAMERA CONFIG ##########

cameras:
  frigate_street_front:
    enabled: True
    ffmpeg:
      hwaccel_args: preset-nvidia-h264
      output_args:
        record: preset-record-generic
      inputs:
        - path: rtsp:https://10.0.0.100:8554/frigate_street_front
          input_args: preset-rtsp-generic
          roles:
            - record
            - detect
    detect:
      enabled: true

  frigate_street_nw:
    enabled: True
    ffmpeg:
      hwaccel_args: preset-nvidia-h264      
      output_args:
        record: preset-record-generic
      inputs:
        - path: rtsp:https://10.0.0.100:8554/frigate_street_nw
          input_args: preset-rtsp-generic
          roles:
            - record
            - detect
    detect:
      enabled: true


  frigate_street_se:
    enabled: True
    ffmpeg:
      hwaccel_args: preset-nvidia-h264      
      output_args:
        record: preset-record-generic
      inputs:
        - path: rtsp:https://10.0.0.100:8554/frigate_street_se
          input_args: preset-rtsp-generic    
          roles:
            - record
            - detect
    detect:
      enabled: true

  frigate_backyard:
    enabled: True
    ffmpeg:
      hwaccel_args: preset-nvidia-h264      
      output_args:
        record: preset-record-generic
      inputs:
        - path: rtsp:https://10.0.0.100:8554/frigate_backyard
          input_args: preset-rtsp-generic
          roles:
            - record
            - detect
    detect:
      enabled: true

Relevant log output

2023-07-27 05:38:12.267286217  [INFO] Starting Frigate...
2023-07-27 05:38:14.454945761  [2023-07-27 05:38:14] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-07-27 05:38:14.497220341  [2023-07-27 05:38:14] peewee_migrate                 INFO    : Starting migrations
2023-07-27 05:38:14.509360514  [2023-07-27 05:38:14] peewee_migrate                 INFO    : There is nothing to migrate
2023-07-27 05:38:14.534655211  [2023-07-27 05:38:14] frigate.app                    INFO    : Output process started: 866
2023-07-27 05:38:14.539646275  [2023-07-27 05:38:14] frigate.app                    INFO    : Camera processor started for frigate_street_front: 869
2023-07-27 05:38:14.545823867  [2023-07-27 05:38:14] frigate.app                    INFO    : Camera processor started for frigate_street_nw: 870
2023-07-27 05:38:14.553883919  [2023-07-27 05:38:14] frigate.app                    INFO    : Camera processor started for frigate_street_se: 871
2023-07-27 05:38:14.560161238  [2023-07-27 05:38:14] frigate.app                    INFO    : Camera processor started for frigate_backyard: 872
2023-07-27 05:38:14.560916236  [2023-07-27 05:38:14] detector.tensorrt              INFO    : Starting detection process: 864
2023-07-27 05:38:14.566928155  [2023-07-27 05:38:14] frigate.app                    INFO    : Capture process started for frigate_street_front: 875
2023-07-27 05:38:14.573195162  [2023-07-27 05:38:14] frigate.app                    INFO    : Capture process started for frigate_street_nw: 878
2023-07-27 05:38:14.578949344  [2023-07-27 05:38:14] frigate.app                    INFO    : Capture process started for frigate_street_se: 881
2023-07-27 05:38:14.585735613  [2023-07-27 05:38:14] frigate.app                    INFO    : Capture process started for frigate_backyard: 884
2023-07-27 05:38:15.476234333  [2023-07-27 05:38:15] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 260, GPU 248 (MiB)
2023-07-27 05:38:15.695521137  [2023-07-27 05:38:15] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 34 MiB
2023-07-27 05:38:15.697033215  [2023-07-27 05:38:15] frigate.detectors.plugins.tensorrt WARNING : Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
2023-07-27 05:38:16.626486368  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +193, GPU +1470, now: CPU 515, GPU 1754 (MiB)
2023-07-27 05:38:16.916109890  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +111, GPU +50, now: CPU 626, GPU 1804 (MiB)
2023-07-27 05:38:16.919309849  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
2023-07-27 05:38:16.927456762  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 591, GPU 1796 (MiB)
2023-07-27 05:38:16.927461745  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +1, GPU +8, now: CPU 592, GPU 1804 (MiB)
2023-07-27 05:38:16.927466361  [2023-07-27 05:38:16] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
2023-07-27 05:41:08.885525340  [2023-07-27 05:41:08] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_backyard. Keeping the 5 most recent segments out of 9 and discarding the rest...
2023-07-27 05:41:08.889404405  [2023-07-27 05:41:08] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_front. Keeping the 5 most recent segments out of 9 and discarding the rest...
2023-07-27 05:41:08.896003680  [2023-07-27 05:41:08] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_nw. Keeping the 5 most recent segments out of 9 and discarding the rest...
2023-07-27 05:41:08.900249221  [2023-07-27 05:41:08] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_se. Keeping the 5 most recent segments out of 9 and discarding the rest...

FFprobe output from your camera

[{"return_code":0,"stderr":"","stdout":{"programs":[],"streams":[{"avg_frame_rate":"0/0","codec_long_name":"H.264/AVC/MPEG-4AVC/MPEG-4part10","height":1944,"width":2592},{"avg_frame_rate":"0/0","bit_rate":"64000","codec_long_name":"PCMA-law/G.711A-law"}]}}]

-> all cameras are the same model and config

Frigate stats

{"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"2.7"},"1":{"cpu":"0.0","mem":"0.0"},"106":{"cpu":"0.0","mem":"0.0"},"112":{"cpu":"1.3","mem":"1.5"},"1154":{"cpu":"0.0","mem":"0.0"},"118":{"cpu":"0.0","mem":"0.0"},"1189":{"cpu":"0.0","mem":"0.0"},"1195":{"cpu":"0.0","mem":"0.1"},"1200":{"cpu":"0.0","mem":"0.0"},"1202":{"cpu":"3.7","mem":"0.1"},"134":{"cpu":"0.0","mem":"0.0"},"135":{"cpu":"0.0","mem":"0.0"},"136":{"cpu":"0.0","mem":"0.0"},"149":{"cpu":"0.0","mem":"0.0"},"15":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"168":{"cpu":"0.0","mem":"0.0"},"187":{"cpu":"0.0","mem":"0.0"},"216":{"cpu":"0.0","mem":"0.0"},"24":{"cpu":"0.0","mem":"0.0"},"241":{"cpu":"0.0","mem":"0.0"},"25":{"cpu":"0.0","mem":"0.0"},"26":{"cpu":"0.0","mem":"0.0"},"263":{"cpu":"0.0","mem":"0.0"},"27":{"cpu":"0.0","mem":"0.0"},"271":{"cpu":"0.0","mem":"0.0"},"28":{"cpu":"0.0","mem":"0.0"},"280":{"cpu":"0.0","mem":"0.0"},"29":{"cpu":"0.0","mem":"0.0"},"290":{"cpu":"0.0","mem":"0.0"},"30":{"cpu":"0.0","mem":"0.0"},"303":{"cpu":"0.0","mem":"0.0"},"31":{"cpu":"0.0","mem":"0.0"},"32":{"cpu":"0.0","mem":"0.0"},"326":{"cpu":"0.0","mem":"0.0"},"348":{"cpu":"0.0","mem":"0.0"},"384":{"cpu":"0.0","mem":"0.0"},"41":{"cpu":"0.0","mem":"0.0"},"42":{"cpu":"0.0","mem":"0.0"},"425":{"cpu":"0.0","mem":"0.0"},"450":{"cpu":"0.0","mem":"0.0"},"475":{"cpu":"0.0","mem":"0.0"},"499":{"cpu":"0.0","mem":"0.0"},"80":{"cpu":"0.0","mem":"0.0"},"81":{"cpu":"0.0","mem":"0.0"},"82":{"cpu":"0.0","mem":"0.0"},"858":{"cpu":"0.0","mem":"0.5"},"863":{"cpu":"0.3","mem":"0.0"},"864":{"cpu":"12.7","mem":"3.4"},"866":{"cpu":"2.0","mem":"0.6"},"869":{"cpu":"13.0","mem":"0.6"},"872":{"cpu":"52.3","mem":"0.7"},"873":{"cpu":"26.3","mem":"0.7"},"874":{"cpu":"0.7","mem":"0.6"},"875":{"cpu":"5.0","mem":"0.6"},"877":{"cpu":"5.3","mem":"0.6"},"879":{"cpu":"0.0","mem":"0.1"},"880":{"cpu":"5.3","mem":"0.6"},"885":{"cpu":"11.3","mem":"0.8"},"886":{"cpu":"11.3","mem":"0.8"},"887":{"cpu":"5.3","mem":"0.6"},"888":{"cpu":"0.0","mem":"0.1"},"889":{"cpu":"10.7","mem":"0.9"},"89":{"cpu":"0.0","mem":"0.1"},"895":{"cpu":"0.0","mem":"0.1"},"896":{"cpu":"12.3","mem":"0.8"},"898":{"cpu":"0.0","mem":"0.1"},"900":{"cpu":"0.0","mem":"0.1"},"MiB":{"cpu":"12239.3","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"average:","mem":"24.76,"}},"detection_fps":22.7,"detectors":{"tensorrt":{"detection_start":0.0,"inference_speed":4.66,"pid":864}},"frigate_backyard":{"camera_fps":5.0,"capture_pid":887,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":896,"pid":874,"process_fps":5.0,"skipped_fps":0.0},"frigate_street_front":{"camera_fps":5.0,"capture_pid":875,"detection_enabled":1,"detection_fps":5.2,"ffmpeg_pid":885,"pid":869,"process_fps":5.0,"skipped_fps":0.0},"frigate_street_nw":{"camera_fps":5.1,"capture_pid":877,"detection_enabled":1,"detection_fps":12.1,"ffmpeg_pid":886,"pid":872,"process_fps":5.1,"skipped_fps":0.0},"frigate_street_se":{"camera_fps":5.1,"capture_pid":880,"detection_enabled":1,"detection_fps":5.4,"ffmpeg_pid":889,"pid":873,"process_fps":5.1,"skipped_fps":0.0},"gpu_usages":{"NVIDIA GeForce GTX 1080":{"gpu":"6 %","mem":"22.2 %"}},"service":{"last_updated":1690430774,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":5335.6,"mount_type":"tmpfs","total":5368.7,"used":33.1},"/media/frigate/clips":{"free":458769.1,"mount_type":"ext4","total":2998506.7,"used":2416951.1},"/media/frigate/recordings":{"free":458769.1,"mount_type":"ext4","total":2998506.7,"used":2416951.1},"/tmp/cache":{"free":1915.6,"mount_type":"tmpfs","total":2000.0,"used":84.4}},"temperatures":{},"uptime":54,"version":"0.12.1-367d724"}}

Operating system

HassOS

Install method

HassOS Addon

Coral version

USB

Network connection

Wired

Camera make and model

Anpviz 5MP

Any other information that may be helpful

No response

@Majestic7979
Copy link
Author

This is the camera details - my camera is basically the same one, but mine has 4x optical zoom. They don't sell it anymore, but it's pointless anyway since I never use the zoom...
https://www.anpvizsecurity.com/products-detail-149850

@Majestic7979
Copy link
Author

image

@Majestic7979
Copy link
Author

image

@Majestic7979
Copy link
Author

image

@Majestic7979
Copy link
Author

image

@Majestic7979
Copy link
Author

image

@Majestic7979
Copy link
Author

2023-07-27 05:38:12.248355587 [INFO] Preparing go2rtc config...
2023-07-27 05:38:12.840348777 [INFO] Starting go2rtc...
2023-07-27 05:38:12.927546476 05:38:12.927 INF go2rtc version 1.2.0 linux/amd64
2023-07-27 05:38:12.927863464 05:38:12.927 INF [api] listen addr=:1984
2023-07-27 05:38:12.928059559 05:38:12.928 INF [rtsp] listen addr=:8554
2023-07-27 05:38:12.928251082 05:38:12.928 INF [srtp] listen addr=:8443
2023-07-27 05:38:12.928434646 05:38:12.928 INF [webrtc] listen addr=:8555
2023-07-27 05:38:22.250947703 [INFO] Starting go2rtc healthcheck service...

@Majestic7979
Copy link
Author

2023-07-27 05:38:12.252644235  [INFO] Starting NGINX...
2023-07-27 05:38:12.787907773  2023/07/27 05:38:12 [error] 133#133: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.211, server: , request: "GET /api/stats HTTP/1.1", upstream: "https://127.0.0.1:5001/stats", host: "10.0.0.100:5000"
2023-07-27 05:38:12.787913838  10.0.0.211 - - [27/Jul/2023:05:38:12 +0100] "GET /api/stats HTTP/1.1" 502 157 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.431646851  10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.436619483  10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.438426169  10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:16.449435910  10.0.0.211 - - [27/Jul/2023:05:38:16 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 2624 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.231424595  10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3213 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.297277909  10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3662 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.308432389  10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3546 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:20.329202143  10.0.0.211 - - [27/Jul/2023:05:38:20 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3741 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:21.036429727  10.0.0.211 - - [27/Jul/2023:05:38:21 +0100] "GET /api/stats HTTP/1.1" 200 3557 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.889390774  10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8561 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.890571543  10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7279 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.891752588  10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9158 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:25.915748783  10.0.0.211 - - [27/Jul/2023:05:38:25 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8396 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:29.059318788  10.0.0.211 - - [27/Jul/2023:05:38:29 +0100] "GET /api/stats HTTP/1.1" 200 3550 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.291292849  10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3223 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.307715284  10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3544 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.311573641  10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3740 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:30.323592817  10.0.0.211 - - [27/Jul/2023:05:38:30 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3664 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.807765027  10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8583 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.831799781  10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9138 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.837743184  10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7293 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:35.857281786  10.0.0.211 - - [27/Jul/2023:05:38:35 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8424 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:37.043264858  10.0.0.211 - - [27/Jul/2023:05:38:37 +0100] "GET /api/stats HTTP/1.1" 200 3593 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.206203918  10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3739 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.222862769  10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3556 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.263576095  10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:40.264607948  10.0.0.211 - - [27/Jul/2023:05:38:40 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3660 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.043740305  10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/stats HTTP/1.1" 200 3627 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.867897063  10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7303 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.879043706  10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8557 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.901132245  10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8455 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:45.903781198  10.0.0.211 - - [27/Jul/2023:05:38:45 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9163 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.201393673  10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3558 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.212774584  10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3745 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.230717375  10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3649 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:50.230722470  10.0.0.211 - - [27/Jul/2023:05:38:50 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3238 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:53.119573612  10.0.0.211 - - [27/Jul/2023:05:38:53 +0100] "GET /api/stats HTTP/1.1" 200 3647 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.852768669  10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8546 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.874669846  10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7308 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.889875630  10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9188 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:38:55.895579911  10.0.0.211 - - [27/Jul/2023:05:38:55 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8459 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.299693496  10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3235 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.307231210  10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3534 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.340042518  10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3652 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:00.340241709  10.0.0.211 - - [27/Jul/2023:05:39:00 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3755 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:01.051734661  10.0.0.211 - - [27/Jul/2023:05:39:01 +0100] "GET /api/stats HTTP/1.1" 200 3615 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.856406955  10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8570 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.916682313  10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7315 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.935457308  10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9180 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:05.939538823  10.0.0.211 - - [27/Jul/2023:05:39:05 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8422 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:09.079743866  10.0.0.211 - - [27/Jul/2023:05:39:09 +0100] "GET /api/stats HTTP/1.1" 200 3583 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.249082932  10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.256962140  10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3542 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.261053412  10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3652 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:10.265379335  10.0.0.211 - - [27/Jul/2023:05:39:10 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3758 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.831576485  10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8581 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.841112385  10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9177 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.861353707  10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8416 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:15.862452710  10.0.0.211 - - [27/Jul/2023:05:39:15 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7302 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:17.027732752  10.0.0.211 - - [27/Jul/2023:05:39:17 +0100] "GET /api/stats HTTP/1.1" 200 3584 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.199703053  10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3250 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.223576289  10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3540 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.243577429  10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3755 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:20.259714764  10.0.0.211 - - [27/Jul/2023:05:39:20 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3665 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.056271979  10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/stats HTTP/1.1" 200 3664 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.851576288  10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9233 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.855568354  10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7314 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.859734345  10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8529 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:25.876347884  10.0.0.211 - - [27/Jul/2023:05:39:25 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8449 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.302785553  10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3546 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.310445539  10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.335745205  10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3768 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:30.365298962  10.0.0.211 - - [27/Jul/2023:05:39:30 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3651 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:33.064522539  10.0.0.211 - - [27/Jul/2023:05:39:33 +0100] "GET /api/stats HTTP/1.1" 200 3598 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.834617599  10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8540 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.857656229  10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7300 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.862026021  10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8440 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:35.883778149  10.0.0.211 - - [27/Jul/2023:05:39:35 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9237 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.210843173  10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3244 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.218080399  10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3563 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.246018290  10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3782 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:40.267585193  10.0.0.211 - - [27/Jul/2023:05:39:40 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3661 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:41.033828433  10.0.0.211 - - [27/Jul/2023:05:39:41 +0100] "GET /api/stats HTTP/1.1" 200 3585 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.875895779  10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8591 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.892384057  10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7415 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.900027351  10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9247 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:45.906613740  10.0.0.211 - - [27/Jul/2023:05:39:45 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8462 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:49.047173785  10.0.0.211 - - [27/Jul/2023:05:39:49 +0100] "GET /api/stats HTTP/1.1" 200 3628 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.237548382  10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3535 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.251151712  10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3667 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.254006903  10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3772 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:50.263504692  10.0.0.211 - - [27/Jul/2023:05:39:50 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3183 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.881816093  10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7229 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.883126887  10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8532 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.891740992  10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8369 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:56.896643805  10.0.0.211 - - [27/Jul/2023:05:39:56 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9247 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:39:57.060453696  10.0.0.211 - - [27/Jul/2023:05:39:57 +0100] "GET /api/stats HTTP/1.1" 200 3617 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.191185760  10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3193 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.195884250  10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3530 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.201606413  10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3784 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:00.228000964  10.0.0.211 - - [27/Jul/2023:05:40:00 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3648 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:05.056632599  10.0.0.211 - - [27/Jul/2023:05:40:05 +0100] "GET /api/stats HTTP/1.1" 200 3598 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.196302668  10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3208 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.219712275  10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3531 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.222178318  10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3774 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:10.238280609  10.0.0.211 - - [27/Jul/2023:05:40:10 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3637 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:13.044757436  10.0.0.211 - - [27/Jul/2023:05:40:13 +0100] "GET /api/stats HTTP/1.1" 200 3583 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.838428741  10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8548 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.887592170  10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7232 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.899749130  10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8387 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:15.907584023  10.0.0.211 - - [27/Jul/2023:05:40:15 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9227 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:17.832151443  10.0.0.211 - - [27/Jul/2023:05:40:17 +0100] "GET /config HTTP/1.1" 200 6641 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:17.879712565  10.0.0.211 - - [27/Jul/2023:05:40:17 +0100] "GET /api/config HTTP/1.1" 200 12623 "https://mydomain.example/config" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:18.163871560  10.0.0.211 - - [27/Jul/2023:05:40:18 +0100] "GET /api/config/raw HTTP/1.1" 200 1002 "https://mydomain.example/config" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:18.431577478  10.0.0.211 - - [27/Jul/2023:05:40:18 +0100] "GET //api/config/schema.json HTTP/1.1" 200 25729 "https://mydomain.example/assets/yaml.worker.bundle.js" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:20.242401440  10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3195 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:20.262206588  10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3788 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:20.262212030  10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3522 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:20.278525963  10.0.0.211 - - [27/Jul/2023:05:40:20 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3653 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:21.043582568  10.0.0.211 - - [27/Jul/2023:05:40:21 +0100] "GET /api/stats HTTP/1.1" 200 3650 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:22.710410903  10.0.0.211 - - [27/Jul/2023:05:40:22 +0100] "GET /api/stats HTTP/1.1" 200 3648 "https://mydomain.example/system" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:25.879279294  10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8542 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:25.899580698  10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7242 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:25.909508582  10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9258 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:25.929518059  10.0.0.211 - - [27/Jul/2023:05:40:25 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8381 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:26.761249705  10.0.0.211 - - [27/Jul/2023:05:40:26 +0100] "GET /api/logs/nginx HTTP/1.1" 200 2532 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:26.777079919  10.0.0.211 - - [27/Jul/2023:05:40:26 +0100] "GET /api/logs/go2rtc HTTP/1.1" 200 258 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:26.800516415  10.0.0.211 - - [27/Jul/2023:05:40:26 +0100] "GET /api/logs/frigate HTTP/1.1" 200 821 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:28.293681829  10.0.0.211 - - [27/Jul/2023:05:40:28 +0100] "GET /api/stats HTTP/1.1" 200 3683 "https://mydomain.example/storage" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:28.509471747  10.0.0.211 - - [27/Jul/2023:05:40:28 +0100] "GET /api/recordings/storage HTTP/1.1" 200 390 "https://mydomain.example/storage" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:29.052124839  10.0.0.211 - - [27/Jul/2023:05:40:29 +0100] "GET /api/stats HTTP/1.1" 200 3683 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.178105819  10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_street_front/latest.jpg?h=87 HTTP/1.1" 200 3218 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.238486528  10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_street_se/latest.jpg?h=87 HTTP/1.1" 200 3541 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.247574161  10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_backyard/latest.jpg?h=87 HTTP/1.1" 200 3781 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:30.299575305  10.0.0.211 - - [27/Jul/2023:05:40:30 +0100] "GET /api/frigate_street_nw/latest.jpg?h=87 HTTP/1.1" 200 3650 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:35.887575575  10.0.0.211 - - [27/Jul/2023:05:40:35 +0100] "GET /api/frigate_street_nw/latest.jpg?h=148 HTTP/1.1" 200 8571 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:35.923575548  10.0.0.211 - - [27/Jul/2023:05:40:35 +0100] "GET /api/frigate_street_front/latest.jpg?h=148 HTTP/1.1" 200 7263 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:36.799577193  10.0.0.211 - - [27/Jul/2023:05:40:36 +0100] "GET /api/frigate_backyard/latest.jpg?h=148 HTTP/1.1" 200 9231 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:36.847569285  10.0.0.211 - - [27/Jul/2023:05:40:36 +0100] "GET /api/frigate_street_se/latest.jpg?h=148 HTTP/1.1" 200 8423 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:37.043577178  10.0.0.211 - - [27/Jul/2023:05:40:37 +0100] "GET /api/stats HTTP/1.1" 200 3618 "-" "HomeAssistant/2023.7.3 aiohttp/3.8.5 Python/3.11" "-"
2023-07-27 05:40:39.968607191  10.0.0.211 - - [27/Jul/2023:05:40:39 +0100] "GET /api/config HTTP/1.1" 200 12623 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"
2023-07-27 05:40:39.970618926  10.0.0.211 - - [27/Jul/2023:05:40:39 +0100] "GET /api/logs/go2rtc HTTP/1.1" 200 258 "https://mydomain.example/logs" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36" "10.0.0.1"

@Majestic7979
Copy link
Author

Host:




server.local system information
SYSTEM INFORMATION     
System hostname server.local (127.0.1.1)
Operating system Debian Linux 12
Webmin version 2.100
Authentic theme version 21.01-2
Time on system Thursday, July 27, 2023 5:48 AM
Kernel and CPU Linux 6.1.0-10-amd64 on x86_64
Processor information 13th Gen Intel(R) Core(TM) i5-13500, 20 cores
CPU temperatures Core 1: 97°CCore 5: 96°CCore 9: 99°CCore 13: 100°CCore 17: 98°CCore 21: 100°CCore 25: 92°CCore 26: 92°CCore 27: 92°CCore 28: 92°CCore 29: 89°CCore 30: 89°CCore 31: 89°CCore 32: 89°C
CPU fans Fan 1: 0 RPMFan 2: 3125 RPMFan 6: 0 RPMFan 7: 0 RPM
Drive temperatures nvme0n1: 67°Csda: 44°Csdb: 43°Csdc: 41°C
System uptime 4 hours, 52 minutes
Running processes 936
CPU load averages 32.26 (1 min) 34.29 (5 mins) 34.86 (15 mins)
Real memory 18.58 GiB used / 12.22 GiB cached / 31.08 GiB total
Virtual memory 7.35 GiB used / 63.99 GiB total
Local disk space 6.54 TiB used / 11.55 TiB free / 18.09 TiB total
Package updates All installed packages are up to date

@Majestic7979
Copy link
Author

I want you to ignore the CPU temps. It's high because ffmpeg is running on another process (immich) via docker to transcode some videos. The issue I have with Frigate not keeping up with frames happens even when the CPU is not stressed. Regardless, if you will insist that this is the issue let me know how to force the frigate container to run as a priority because my security system must always be prioritized above all else.

@blakeblackshear
Copy link
Owner

blakeblackshear commented Jul 27, 2023

Is the disk where recordings are stored on the local machine or are you storing them on a network share?

You can turn on debug logging to see the copy times for moving recordings:

logger:
  logs:
    frigate.record: debug

If the copy times are fast, then in all other instances this has happened from the CPU load on the machine being too high for Frigate to have the resources to keep up.

I do not know of a way to specify priority in HassOS for an addon.

Alternatively, you can use motion mode instead of all mode for record so fewer segments are copied over. This has helped several users.

Anything you can do to reduce unnecessary processing for frigate will help. You are running detect at a very high resolution which is not necessary for most users, but perhaps you are trying to detect people who are very very far away. If not, using a lower resolution stream or downscaling should help. Make sure you have setup motion masks so it isn't wasting cycles running analysis on things like timestamps.

@Majestic7979
Copy link
Author

Majestic7979 commented Jul 27, 2023

Is the disk where recordings are stored on the local machine or are you storing them on a network share?

You can turn on debug logging to see the copy times for moving recordings:

logger:
  logs:
    frigate.record: debug

If the copy times are fast, then in all other instances this has happened from the CPU load on the machine being too high for Frigate to have the resources to keep up.

I do not know of a way to specify priority in HassOS for an addon.

Alternatively, you can use motion mode instead of all mode for record so fewer segments are copied over. This has helped several users.

Anything you can do to reduce unnecessary processing for frigate will help. You are running detect at a very high resolution which is not necessary for most users, but perhaps you are trying to detect people who are very very far away. If not, using a lower resolution stream or downscaling should help. Make sure you have setup motion masks so it isn't wasting cycles running analysis on things like timestamps.

It's a SATA disk in the host device.

I get this issue even when the CPU is in the low percent of usage. I knew this was going to come up... It may be that, I'm not going to say "it's not that", without data neither of us don't know, so it might be. But it happens even when there's no high CPU utilization. Nick told me to increase shm, he said that was the issue. But it didn't fix, you can see my shm is quite high.
As per above form answer I don't use HassOS, I run directly on Docker on a Debian 12 system.

I cannot use motion for recording, It's simply not reliable enough and the police said they need an hour before and an hour after any incident and if there is any skipping they cannot accept the evidence.

The detect documentation said that the resolution must be native from the camera otherwise the CPU needs to scale the image which consumes more CPU. I noticed this answer in several support tickets, to keep the detect at the original res.

Is the issue related to motion detection? Or is the issue related to recording?

I can try motion masks but the whole point of this project is smart detection isn't it... I mean nobody can say my system is slow, it's better than most people's main personal computers!!

I used to run on a 8GB DDR3 Intel Core i7 3770T processor without GPU (apart from built-in)
Now I run on a 32GB DDR5 Intel Core i5 13500 which is latest gen with many more cores and a relatively beefy GTX1080 GPU and it can't keep up.... Something is not right. This is what I said in my "opinion" - Frigate is focusing a lot on delivering features and skipping on stability, meaning the footage is unusable when it's most required to prosecute a criminal. Honestly, please rethink the approach here. I've seen go2rtc get added with added complexity and added bugs... meanwhile we're seeing a system of this level struggle!? It's not a criticism it's just a fact that's happening right now. Why not focus on multiprocessing/GPU processing? Apparently even with a GPU the CPU is still used quite a lot for some tasks like motion. That should not be happening if there is such a beefy GPU in the host.

But that was digressing. I just need support for the issue right now.

In terms of the debug logging, before I enable I want to know - why are you asking me to enable debug logging (what specifically do you need to see from it)?

Oh and I know I shouldn't expect anyone to assume but it goes without saying I appreciate your time and effort in building this application and supporting others and myself. Big thank you. Never think I am criticizing you.

@NickM-27
Copy link
Sponsor Collaborator

The detect documentation said that the resolution must be native from the camera otherwise the CPU needs to scale the image which consumes more CPU. I noticed this answer in several support tickets, to keep the detect at the original res.

This is no longer true, if you use ffmpeg presets for hwaccel (like you are) then the scaling will be done on the GPU instead so it is not a problem.

Frigate is focusing a lot on delivering features and skipping on stability, meaning the footage is unusable when it's most required to prosecute a criminal. Honestly, please rethink the approach here. I've seen go2rtc get added with added complexity and added bugs... meanwhile we're seeing a system of this level struggle!? It's not a criticism it's just a fact that's happening right now. Why not focus on multiprocessing/GPU processing?

That is objectively false. You can look at the commits for the upcoming 0.13 release, there have been many multiprocessing improvements for recording management and otherwise. But for now the recommendations are made for the current version, which is mainly to reduce the amount of segments that are kept so the recordings maintainer has less work to do.

In terms of the debug logging, before I enable I want to know - why are you asking me to enable debug logging (what specifically do you need to see from it)?

The record debug logging will show the amount of time it takes to move the recording segments from cache to the storage drive, which is crucial to understanding where the slow down is.

@NickM-27
Copy link
Sponsor Collaborator

Also just wanted to confirm, the /db/ volume is not on the Sata SSD correct?

@blakeblackshear
Copy link
Owner

Other users experiencing this problem have had dozens of cameras. I don't think we have seen anyone else run into this issue with as few cameras as you.

@Majestic7979
Copy link
Author

Hey Nick.

What resolution do you suggest me to use then?

Remember I'm talking about what we have currently, up to 0.12. I can't say anything about something I have not yet used... so until 0.12 what I said is at least partly applicable. Again please never take my comments as criticism. I am merely saying that I feel priority is being given to features rather than stability.

I will enable the debug logging now, thank you.

The /db/ is on the SATA drive because I run my container with a bind mount, and the mount is stored in the /home partition which is in a 14TB mirroring RAID.

This is my docker CLI:

docker create --name frigate --restart=always -p 5000:5000 --gpus=all --cap-add=CAP_PERFMON --mount type=tmpfs,target=/tmp/cache,tmpfs-size=2000000000 --shm-size=5g --device /dev/bus/usb:/dev/bus/usb --device /dev/dri/renderD128 -v /cctv:/media/frigate -v /home/user1/dockerbindmounts/frigate/config.yml:/config/config.yml -v /home/user1/dockerbindmounts/frigate/db/:/db/ -v /etc/localtime:/etc/localtime:ro -v /home/user1/trt-models:/trt-models -e FRIGATE_RTSP_PASSWORD='removedforprivacy' -e NVIDIA_VISIBLE_DEVICES=all -e NVIDIA_DRIVER_CAPABILITIES=compute,video,utility -e YOLO_MODELS=yolov7-tiny-416 ghcr.io/blakeblackshear/frigate:0.12.1-tensorrt &&  docker start frigate

@Majestic7979
Copy link
Author

Other users experiencing this problem have had dozens of cameras. I don't think we have seen anyone else run into this issue with as few cameras as you.

Right and that's what is making me scratch my head.

@Majestic7979
Copy link
Author

For info I run a 3TB WD Red Pro drive for my footage, it's mounted on /cctv

@Majestic7979
Copy link
Author

To make absolutely clear I store my footage in a separate SATA drive (3TB WD Red Pro) and my frigate db is in the main RAID array (14TB). The DB and the frigate config/db are in separate storage drives therefore read/write are separate...

@NickM-27
Copy link
Sponsor Collaborator

What resolution do you suggest me to use then?

Looks like your cameras are 4:3 so you could try

detect:
  width: 1280
  height: 960

and see if the objects are detected as you would like. Without seeing the camera images it is difficult to suggest fully because smaller objects that are farther away of course require more resolution to be detected reliably. I run all of my cameras with a height of 720 (and the 4:3 cameras with a height of 960) and have not had any issues detection people far away from the camera.

Remember I'm talking about what we have currently, up to 0.12. I can't say anything about something I have not yet used... so until 0.12 what I said is at least partly applicable. Again please never take my comments as criticism. I am merely saying that I feel priority is being given to features rather than stability.

I would suggest that there were a number of stability improvements in 0.12 as well, but I understand where you're coming from and of course you wouldn't know about the commits in the unreleased version.

The to-do list for 0.13 is getting closer so a beta should not be too far away.

Another thing that could be tried here given that you require all recording instead of motion recording is reducing your cameras bit rate so there is less data for the recordings maintainer to move.

I will enable the debug logging now, thank you.

Great, the values in general will be helpful but it will be mostly helpful to see what the copy times are after the segment cache errors start occurring.

@Majestic7979
Copy link
Author

Majestic7979 commented Jul 27, 2023

Cam image:
[removed for privacy]

Thanks, that's an empathetic response! I really appreciate it!!! I am doing all I can to try to help you help me, and I know you're a stellar person here with your support. It's really nice when someone recognizes comments and takes it on board, understanding that the intentions are always good.

The problem with reducing the bitrate is that it will likely pixellate... this will greatly affect number plate in vehicles especially at night. But the cameras are all feeding directly into a Ubiquiti USW-24-PoE switch which isn't even fully utilized, each camera obviously using their own cable and going into their specific switch port. There is no "in between", it connects directly to my main switch, which the server is also directly connected into.

My cameras are outputting to MediaMTX (rtsp-simple-server) and I input the proxied stream into Frigate. If I input the MediaMTX proxied stream for any of the cams into VLC I get a "perfect" stream. I put perfect into quotes because the audio, for some reason, is slow-mo. You know when you record someone speaking and then you slow-mo and the person sounds like their voice is deeper and slower? This is happening and I don't know why. The video itself is real-time, there is a timestamp as you can see above and it's playing back second to second as expected. I'll now post a new log with debug in a separate comment

@Majestic7979
Copy link
Author

2023-07-27 14:51:24.068200519  [INFO] Starting Frigate...
2023-07-27 14:51:25.991688431  [2023-07-27 14:51:25] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-07-27 14:51:26.035304264  [2023-07-27 14:51:26] peewee_migrate                 INFO    : Starting migrations
2023-07-27 14:51:26.071227368  [2023-07-27 14:51:26] peewee_migrate                 INFO    : There is nothing to migrate
2023-07-27 14:51:26.101592142  [2023-07-27 14:51:26] frigate.app                    INFO    : Output process started: 865
2023-07-27 14:51:26.103897208  [2023-07-27 14:51:26] detector.tensorrt              INFO    : Starting detection process: 864
2023-07-27 14:51:26.128042669  [2023-07-27 14:51:26] frigate.app                    INFO    : Camera processor started for frigate_street_front: 869
2023-07-27 14:51:26.128046642  [2023-07-27 14:51:26] frigate.app                    INFO    : Camera processor started for frigate_street_nw: 870
2023-07-27 14:51:26.128049471  [2023-07-27 14:51:26] frigate.app                    INFO    : Camera processor started for frigate_street_se: 871
2023-07-27 14:51:26.128051525  [2023-07-27 14:51:26] frigate.app                    INFO    : Camera processor started for frigate_backyard: 872
2023-07-27 14:51:26.141239340  [2023-07-27 14:51:26] frigate.app                    INFO    : Capture process started for frigate_street_front: 875
2023-07-27 14:51:26.146591593  [2023-07-27 14:51:26] frigate.app                    INFO    : Capture process started for frigate_street_nw: 878
2023-07-27 14:51:26.153775457  [2023-07-27 14:51:26] frigate.app                    INFO    : Capture process started for frigate_street_se: 882
2023-07-27 14:51:26.159047702  [2023-07-27 14:51:26] frigate.app                    INFO    : Capture process started for frigate_backyard: 885
2023-07-27 14:51:26.813366819  [2023-07-27 14:51:26] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 248 (MiB)
2023-07-27 14:51:26.978411519  [2023-07-27 14:51:26] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 34 MiB
2023-07-27 14:51:26.979692337  [2023-07-27 14:51:26] frigate.detectors.plugins.tensorrt WARNING : Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
2023-07-27 14:51:27.364548466  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +193, GPU +74, now: CPU 501, GPU 358 (MiB)
2023-07-27 14:51:27.557194923  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +110, GPU +46, now: CPU 611, GPU 404 (MiB)
2023-07-27 14:51:27.559115628  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
2023-07-27 14:51:27.565314022  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 577, GPU 396 (MiB)
2023-07-27 14:51:27.565484698  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 577, GPU 404 (MiB)
2023-07-27 14:51:27.565488372  [2023-07-27 14:51:27] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
2023-07-27 14:51:44.271330251  [2023-07-27 14:51:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.27.mp4 in 2.8024659156799316 seconds.
2023-07-27 14:51:47.419609909  [2023-07-27 14:51:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.27.mp4 in 2.8160011768341064 seconds.
2023-07-27 14:51:50.261774582  [2023-07-27 14:51:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.27.mp4 in 2.7169740200042725 seconds.
2023-07-27 14:51:53.223402149  [2023-07-27 14:51:53] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.27.mp4 in 2.8212969303131104 seconds.
2023-07-27 14:51:56.121896583  [2023-07-27 14:51:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.36.mp4 in 2.7229220867156982 seconds.
2023-07-27 14:51:59.070124450  [2023-07-27 14:51:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.36.mp4 in 2.8004510402679443 seconds.
2023-07-27 14:52:01.971288144  [2023-07-27 14:52:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.36.mp4 in 2.7553539276123047 seconds.
2023-07-27 14:52:05.733478023  [2023-07-27 14:52:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.37.mp4 in 3.5945658683776855 seconds.
2023-07-27 14:52:08.670631442  [2023-07-27 14:52:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.46.mp4 in 2.7888638973236084 seconds.
2023-07-27 14:52:11.538703695  [2023-07-27 14:52:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.46.mp4 in 2.696091890335083 seconds.
2023-07-27 14:52:14.378159551  [2023-07-27 14:52:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.46.mp4 in 2.714629888534546 seconds.
2023-07-27 14:52:17.266218893  [2023-07-27 14:52:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.47.mp4 in 2.741323947906494 seconds.
2023-07-27 14:52:20.299887659  [2023-07-27 14:52:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/51.56.mp4 in 2.8629918098449707 seconds.
2023-07-27 14:52:23.280067169  [2023-07-27 14:52:23] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.06.mp4 in 2.8107500076293945 seconds.
2023-07-27 14:52:26.167934835  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/51.56.mp4 in 2.734575033187866 seconds.
2023-07-27 14:52:26.336420434  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Start expire recordings (new).
2023-07-27 14:52:26.336423851  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Start deleted cameras.
2023-07-27 14:52:26.919573876  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Expiring 0 recordings
2023-07-27 14:52:26.923573589  [2023-07-27 14:52:26] frigate.record                 DEBUG   : End deleted cameras.
2023-07-27 14:52:26.923578160  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Start all cameras.
2023-07-27 14:52:26.923580102  [2023-07-27 14:52:26] frigate.record                 DEBUG   : Start camera: frigate_street_front.
2023-07-27 14:52:29.056176497  [2023-07-27 14:52:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.06.mp4 in 2.7186970710754395 seconds.
2023-07-27 14:52:32.019877456  [2023-07-27 14:52:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/51.56.mp4 in 2.780247926712036 seconds.
2023-07-27 14:52:35.067947513  [2023-07-27 14:52:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.06.mp4 in 2.833055019378662 seconds.
2023-07-27 14:52:38.012625476  [2023-07-27 14:52:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/51.57.mp4 in 2.7677149772644043 seconds.
2023-07-27 14:52:40.893284633  [2023-07-27 14:52:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.07.mp4 in 2.7566659450531006 seconds.
2023-07-27 14:52:43.747509775  [2023-07-27 14:52:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.16.mp4 in 2.7001211643218994 seconds.
2023-07-27 14:52:46.693283952  [2023-07-27 14:52:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.27.mp4 in 2.7567219734191895 seconds.
2023-07-27 14:52:49.649205452  [2023-07-27 14:52:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.16.mp4 in 2.7733681201934814 seconds.
2023-07-27 14:52:52.815865164  [2023-07-27 14:52:52] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.26.mp4 in 2.766500949859619 seconds.
2023-07-27 14:52:55.708985391  [2023-07-27 14:52:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.16.mp4 in 2.753082036972046 seconds.
2023-07-27 14:52:58.697371241  [2023-07-27 14:52:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.26.mp4 in 2.7924540042877197 seconds.
2023-07-27 14:53:01.773608922  [2023-07-27 14:53:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.17.mp4 in 2.83032488822937 seconds.
2023-07-27 14:53:04.880077904  [2023-07-27 14:53:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.27.mp4 in 2.8705949783325195 seconds.
2023-07-27 14:53:07.837504876  [2023-07-27 14:53:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.37.mp4 in 2.772428035736084 seconds.
2023-07-27 14:53:10.835078066  [2023-07-27 14:53:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.47.mp4 in 2.780242919921875 seconds.
2023-07-27 14:53:13.794528070  [2023-07-27 14:53:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.36.mp4 in 2.7508840560913086 seconds.
2023-07-27 14:53:16.815387919  [2023-07-27 14:53:16] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.46.mp4 in 2.7242228984832764 seconds.
2023-07-27 14:53:20.046110795  [2023-07-27 14:53:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.36.mp4 in 3.063145160675049 seconds.
2023-07-27 14:53:23.023597905  [2023-07-27 14:53:23] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.46.mp4 in 2.789849042892456 seconds.
2023-07-27 14:53:25.665285688  [2023-07-27 14:53:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.37.mp4 in 2.4909911155700684 seconds.
2023-07-27 14:53:28.683837655  [2023-07-27 14:53:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.46.mp4 in 2.7974019050598145 seconds.
2023-07-27 14:53:31.696797664  [2023-07-27 14:53:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/52.57.mp4 in 2.8094191551208496 seconds.
2023-07-27 14:53:34.678908144  [2023-07-27 14:53:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.07.mp4 in 2.807781934738159 seconds.
2023-07-27 14:53:37.700928489  [2023-07-27 14:53:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.17.mp4 in 2.8578808307647705 seconds.
2023-07-27 14:53:40.810218748  [2023-07-27 14:53:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/52.56.mp4 in 2.919498920440674 seconds.
2023-07-27 14:53:43.785736486  [2023-07-27 14:53:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.06.mp4 in 2.787822961807251 seconds.
2023-07-27 14:53:46.896286987  [2023-07-27 14:53:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.16.mp4 in 2.8843400478363037 seconds.
2023-07-27 14:53:49.849318334  [2023-07-27 14:53:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/52.57.mp4 in 2.8049111366271973 seconds.
2023-07-27 14:53:54.355680230  [2023-07-27 14:53:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.07.mp4 in 4.337353944778442 seconds.
2023-07-27 14:53:59.343875193  [2023-07-27 14:53:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.17.mp4 in 4.821834087371826 seconds.
2023-07-27 14:54:04.400656221  [2023-07-27 14:54:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/52.56.mp4 in 4.908252954483032 seconds.
2023-07-27 14:54:09.597482962  [2023-07-27 14:54:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.06.mp4 in 5.061578989028931 seconds.
2023-07-27 14:54:14.727572184  [2023-07-27 14:54:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.16.mp4 in 4.95068883895874 seconds.
2023-07-27 14:54:18.858567075  [2023-07-27 14:54:18] frigate.record                 DEBUG   : Expiring 192 recordings
2023-07-27 14:54:19.030255548  [2023-07-27 14:54:19] frigate.record                 DEBUG   : End camera: frigate_street_front.
2023-07-27 14:54:19.030336823  [2023-07-27 14:54:19] frigate.record                 DEBUG   : Start camera: frigate_street_nw.
2023-07-27 14:54:19.209223005  [2023-07-27 14:54:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.27.mp4 in 4.3155388832092285 seconds.
2023-07-27 14:54:22.139087749  [2023-07-27 14:54:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.37.mp4 in 2.776745080947876 seconds.
2023-07-27 14:54:27.209055086  [2023-07-27 14:54:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.47.mp4 in 4.884838819503784 seconds.
2023-07-27 14:54:32.727577331  [2023-07-27 14:54:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/53.57.mp4 in 4.9430060386657715 seconds.
2023-07-27 14:54:37.911973178  [2023-07-27 14:54:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.26.mp4 in 5.032839059829712 seconds.
2023-07-27 14:54:43.181262654  [2023-07-27 14:54:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.36.mp4 in 5.1069581508636475 seconds.
2023-07-27 14:54:48.060652981  [2023-07-27 14:54:48] frigate.record                 DEBUG   : Expiring 190 recordings
2023-07-27 14:54:48.115701657  [2023-07-27 14:54:48] frigate.record                 DEBUG   : End camera: frigate_street_nw.
2023-07-27 14:54:48.115706041  [2023-07-27 14:54:48] frigate.record                 DEBUG   : Start camera: frigate_street_se.
2023-07-27 14:54:48.141978684  [2023-07-27 14:54:48] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.46.mp4 in 4.81535005569458 seconds.
2023-07-27 14:54:51.013401169  [2023-07-27 14:54:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/53.56.mp4 in 2.715502977371216 seconds.
2023-07-27 14:54:55.920729949  [2023-07-27 14:54:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.27.mp4 in 4.715717077255249 seconds.
2023-07-27 14:55:01.098977707  [2023-07-27 14:55:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.37.mp4 in 4.994410991668701 seconds.
2023-07-27 14:55:06.160359378  [2023-07-27 14:55:06] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.47.mp4 in 4.908405065536499 seconds.
2023-07-27 14:55:11.219192641  [2023-07-27 14:55:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/53.57.mp4 in 4.902930021286011 seconds.
2023-07-27 14:55:15.146357792  [2023-07-27 14:55:15] frigate.record                 DEBUG   : Expiring 173 recordings
2023-07-27 14:55:15.222282726  [2023-07-27 14:55:15] frigate.record                 DEBUG   : End camera: frigate_street_se.
2023-07-27 14:55:15.222387908  [2023-07-27 14:55:15] frigate.record                 DEBUG   : Start camera: frigate_backyard.
2023-07-27 14:55:15.498984302  [2023-07-27 14:55:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.26.mp4 in 4.104842185974121 seconds.
2023-07-27 14:55:18.325911082  [2023-07-27 14:55:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.36.mp4 in 2.706555128097534 seconds.
2023-07-27 14:55:23.467614561  [2023-07-27 14:55:23] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.46.mp4 in 4.991831064224243 seconds.
2023-07-27 14:55:28.530655564  [2023-07-27 14:55:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/53.56.mp4 in 4.9489710330963135 seconds.
2023-07-27 14:55:28.561354799  [2023-07-27 14:55:28] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_backyard. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:28.565064279  [2023-07-27 14:55:28] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_front. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:28.567689923  [2023-07-27 14:55:28] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_nw. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:28.570692819  [2023-07-27 14:55:28] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_se. Keeping the 5 most recent segments out of 8 and discarding the rest...
2023-07-27 14:55:33.305559493  [2023-07-27 14:55:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/54.36.mp4 in 4.606783151626587 seconds.
2023-07-27 14:55:38.091694580  [2023-07-27 14:55:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/54.46.mp4 in 4.640094995498657 seconds.
2023-07-27 14:55:43.146840524  [2023-07-27 14:55:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/54.56.mp4 in 4.912054061889648 seconds.
2023-07-27 14:55:44.605703963  [2023-07-27 14:55:44] frigate.record                 DEBUG   : Expiring 192 recordings
2023-07-27 14:55:44.705574828  [2023-07-27 14:55:44] frigate.record                 DEBUG   : End camera: frigate_backyard.
2023-07-27 14:55:44.705579297  [2023-07-27 14:55:44] frigate.record                 DEBUG   : End all cameras.
2023-07-27 14:55:44.705581368  [2023-07-27 14:55:44] frigate.record                 DEBUG   : End expire recordings (new).
2023-07-27 14:55:44.705582996  [2023-07-27 14:55:44] frigate.record                 DEBUG   : Start expire files (legacy).
2023-07-27 14:55:44.751749462  [2023-07-27 14:55:44] frigate.record                 DEBUG   : Oldest recording in the db: 1689601642.515418
2023-07-27 14:55:48.178487178  [2023-07-27 14:55:48] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/55.06.mp4 in 4.88518500328064 seconds.
2023-07-27 14:55:51.890108621  [2023-07-27 14:55:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_backyard/55.16.mp4 in 3.5519468784332275 seconds.
2023-07-27 14:55:56.885556220  [2023-07-27 14:55:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/54.36.mp4 in 4.819516897201538 seconds.
2023-07-27 14:56:01.939756131  [2023-07-27 14:56:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/54.46.mp4 in 4.904176950454712 seconds.
2023-07-27 14:56:06.808378756  [2023-07-27 14:56:06] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/54.56.mp4 in 4.73555588722229 seconds.
2023-07-27 14:56:12.329512784  [2023-07-27 14:56:12] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/55.07.mp4 in 5.364140033721924 seconds.
2023-07-27 14:56:18.414092681  [2023-07-27 14:56:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_front/55.17.mp4 in 5.933212041854858 seconds.
2023-07-27 14:56:24.186002145  [2023-07-27 14:56:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/54.36.mp4 in 5.628190040588379 seconds.
2023-07-27 14:56:30.308024214  [2023-07-27 14:56:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/54.46.mp4 in 5.9718639850616455 seconds.
2023-07-27 14:56:35.452113307  [2023-07-27 14:56:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/54.56.mp4 in 4.97012996673584 seconds.
2023-07-27 14:56:40.990185255  [2023-07-27 14:56:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/55.06.mp4 in 5.417521953582764 seconds.
2023-07-27 14:56:46.175705449  [2023-07-27 14:56:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_nw/55.16.mp4 in 4.98787784576416 seconds.
2023-07-27 14:56:51.373647527  [2023-07-27 14:56:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/54.36.mp4 in 5.038317918777466 seconds.
2023-07-27 14:56:54.940395663  [2023-07-27 14:56:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/54.46.mp4 in 3.3942999839782715 seconds.
2023-07-27 14:56:58.543646121  [2023-07-27 14:56:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/54.56.mp4 in 3.4380130767822266 seconds.
2023-07-27 14:57:02.081852655  [2023-07-27 14:57:02] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-07-27/13/frigate_street_se/55.06.mp4 in 3.396026134490967 seconds.

@NickM-27
Copy link
Sponsor Collaborator

Based on that camera image I think 1280x960 for detect should work well, you can always verify by watching the debug live view.

As far as the debug logs go, it does look like the storage is the problem. The storage is taking a decent amount of time to move segments. Given that you have 4 cameras, there are 4 10-second segments created every 10 seconds. Even if your 4 cameras have the fastest shown times of 3 seconds to move each segment to the storage that is still 12 seconds total to move 1 segment per camera. Which means that your recording storage is not able to keep up with everything that is being written.

I definitely think the multiprocessing and other improvements in 0.13 will greatly improve this, but it seems that in your case storage is the limiting factor right now.

@Majestic7979
Copy link
Author

Based on that camera image I think 1280x960 for detect should work well, you can always verify by watching the debug live view.

As far as the debug logs go, it does look like the storage is the problem. The storage is taking a decent amount of time to move segments. Given that you have 4 cameras, there are 4 10-second segments created every 10 seconds. Even if your 4 cameras have the fastest shown times of 3 seconds to move each segment to the storage that is still 12 seconds total to move 1 segment per camera. Which means that your recording storage is not able to keep up with everything that is being written.

I definitely think the multiprocessing and other improvements in 0.13 will greatly improve this, but it seems that in your case storage is the limiting factor right now.

Thanks, having read that, all I can say is that on 0.11 and before these issues weren't really present... Or if they were, I just never noticed. The only time I noticed skipping was when I used a ffmpeg format that included the audio track, but Frigate just does not like it from my cam, so I drop it. I use the generic preset.

I suggest... How about we look into my camera output in more detail and see if we can debug what's being sent to Frigate? Do you want a raw output stream? I can send you a link from cloud storage for debug purposes. What do you propose other than the storage? It's kinda hard to accept that a hard drive with over 100MBps write capacity can't keep up with videos that are at most 8MB per file. Even with four cameras it would be around 40MBps at most... that drive is exclusively for Frigate footage.

SMART output follows in next comment

@Majestic7979
Copy link
Author

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-10-amd64] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-xxxxxxxxxx
LU WWN Device Id: 5 0014ee xxxxxx
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database 7.3/5319
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Jul 27 16:53:55 2023 BST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(40500) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 406) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x703d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   191   175   021    Pre-fail  Always       -       5450
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       852
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   001   001   000    Old_age   Always       -       78776
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       775
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       226
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       625
194 Temperature_Celsius     0x0022   110   103   000    Old_age   Always       -       40
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     52260         -
# 2  Short offline       Completed without error       00%     52236         -
# 3  Short offline       Completed without error       00%     46009         -
# 4  Short offline       Completed without error       00%     45985         -
# 5  Short offline       Completed without error       00%     45961         -
# 6  Short offline       Completed without error       00%     45937         -
# 7  Short offline       Completed without error       00%     45913         -
# 8  Short offline       Completed without error       00%     45889         -
# 9  Short offline       Completed without error       00%     45865         -
#10  Short offline       Completed without error       00%     45841         -
#11  Short offline       Completed without error       00%     45817         -
#12  Short offline       Completed without error       00%     45793         -
#13  Short offline       Completed without error       00%     45769         -
#14  Short offline       Completed without error       00%     45745         -
#15  Short offline       Completed without error       00%     45721         -
#16  Short offline       Completed without error       00%     45697         -
#17  Short offline       Completed without error       00%     45673         -
#18  Short offline       Completed without error       00%     45649         -
#19  Short offline       Completed without error       00%     45625         -
#20  Extended offline    Completed without error       00%     43179         -
#21  Short offline       Completed without error       00%      8992         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

@Majestic7979
Copy link
Author

image

@NickM-27
Copy link
Sponsor Collaborator

Thanks, having read that, all I can say is that on 0.11 and before these issues weren't really present... Or if they were, I just never noticed. The only time I noticed skipping was when I used a ffmpeg format that included the audio track, but Frigate just does not like it from my cam, so I drop it. I use the generic preset.

Frigate 0.11 did not have these warning logs so there would be no way to know except notice the recording segment wasn't there.

It's kinda hard to accept that a hard drive with over 100MBps write capacity can't keep up with videos that are at most 8MB per file. Even with four cameras it would be around 40MBps at most... that drive is exclusively for Frigate footage.

For one it's not constant write it's separate writes for each segment. The timing logs are very simple around the ffmpeg call to optimize the segment metadata and write to storage so there's no room for error in that calculation.

Can you show your docker compose? One explanation would be if /tmp/cache isn't using RAM but is using disk instead.

@Majestic7979
Copy link
Author

image
I cannot do write test because I'm on the live system

@NickM-27
Copy link
Sponsor Collaborator

Odd, definitely seems like the cameras are sending bad data. I'm surprised AAC only has an option for 32000, maybe try PCM restart mediamtx and see what happens in Vic and frigate

@Majestic7979
Copy link
Author

When you say try PCM do you mean G711/G711A? Actually, with AAC the camera UI shows only 16k, not 32. But on VLC it is reported as 32. It's Chinese.... half-assed clearly. Supposed to be Hik clones. The image sensors they use are actually good, the Police has commented on how shar the image looks - except on h265... I have seen a difference. H264 is way sharper on these. I paid £44 per camera. It was a new brand seller on Amazon at the time, when I still used Amazon. So I got it for a bargain. Optical zoom with autofocus... No PTZ sadly, but the zoom allows for more flexible placement. But clearly the firmware is crap. I am sure that it might be possible to find out exactly what is being reported regardless of what the camera says, and manually set ffmpeg/frigate to deal with it accordingly? We will see. I am severely sleep deprived. I caught a 2nd burglary this week. 2 in one week. It's stressful. I really depend on Frigate, and appreciate your efforts. It's not about money. It's about not supporting China. I don't want to buy their NVRs, seems like every NVR is made there. I stopped buying Chinese goods a few years ago. Only as a last resort. Frigate is open, and you guys are doing a great job from the US. I want to see Frigate take off the way Home Assistant did, and I believe it will happen. We need a good, western-based open source Linux NVR system. Keep at it. I'll get back to you.

@NickM-27
Copy link
Sponsor Collaborator

Try G711A first would be my suggestion

@Majestic7979
Copy link
Author

Majestic7979 commented Aug 1, 2023

All good now. You won't believe... seems like the audio issue was linked to the frame dropping... Once the h265 got enabled and the frames stopped dropping so much, looks like the audio started working. So it wasn't anything to do with changing the codecs but something in the encoding being affected by dropped frames. Actually G711A is what stopped the "static". AAC from the cam seems to be broken. But the audio not coming through at all was the frame drop issue. Shrug. It's all good. I'm going to leave this open, the hard drive should be delivered today, I want to confirm with you real quick, this is what I plan to do:

  1. stop the frigate container
  2. install both hdds in my server
  3. cp * from the old hdd to the new hdd
  4. edit fstab to mount the new hdd to the existing /cctv path
  5. remove old hdd
  6. spin the container back up
  7. PROFIT? Will I see all the history? Or is there something else I need to do?

Once the hard drive is installed, we can then confirm the issue (which I saw added by blake on 0.13 tracker). You said that you use an SSD as cache to speed things up... Actually, I remembered I have a Crucial M4 I am not using, so I am thinking, could you provide me the guide you used to create your cached storage solution with SSD and HDD? I am using Debian 12. I will have a go at this.... this CCTV is being a life saver this month. Two burglaries and a neighbor dispute...

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Aug 1, 2023

I'm going to leave this open, the hard drive should be delivered today, I want to confirm with you real quick, this is what I plan to do:

  1. stop the frigate container
  2. install both hdds in my server
  3. cp * from the old hdd to the new hdd
  4. edit fstab to mount the new hdd to the existing /cctv path
  5. remove old hdd
  6. spin the container back up
  7. PROFIT? Will I see all the history? Or is there something else I need to do?

That should be it

Once the hard drive is installed, we can then confirm the issue (which I saw added by blake on 0.13 tracker).

That was added before, when users were seeing this issue running 0.13 after recordings maintainer was moved to its own process. Since then more improvements and optimizations have been added so as long as a users storage can keep up, Frigate should not have any issues.

You said that you use an SSD as cache to speed things up... Actually, I remembered I have a Crucial M4 I am not using, so I am thinking, could you provide me the guide you used to create your cached storage solution with SSD and HDD? I am using Debian 12. I will have a go at this.... this CCTV is being a life saver this month. Two burglaries and a neighbor dispute...

I run unRAID OS which uses fuse.fs to make multiple storage mediums into a single "share" for frigate to write to. It then has a mover which moves files from the SSD to the HDD when it gets above 60%. I am not sure how difficult it would be to do in a separate OS since it is built in to unRAID

@Majestic7979
Copy link
Author

I'm currently transferring the data to the new HDD. The audio is working but it's also not working - while there is audio, it seems like it's slowed down, it's not 1x speed but seems like 0.5x speed, so the voice is distorted, sounds like a slow-speaking very low tone voice... like when you use a sound editing app and slow the audio. That's what is happening. So it's pretty useless. Any ideas? I can send you a sample clip if that helps.

@Majestic7979
Copy link
Author

New drive installed, here's some data from it before formatting, only for reference:
image

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Aug 2, 2023

To me it just seems that the cameras audio is bugged quite a bit which is making things difficult. You could try using preset-rtsp-restream for the input_args since it is a restream. I have one other idea but better to try that first

@Majestic7979
Copy link
Author

So the audio saga comes to an end. The camera firmware is dumb. It seems to report 16k but sends data at 8k sampling rate anyway. That's why it was "slowed by half". I set the cam to 8k and it's now absolutely clear. So - now I have video AND audio working at full res. If you want me to send you my exact specs to add my brand and model to the documentation please let me know.

@Majestic7979
Copy link
Author

Majestic7979 commented Aug 3, 2023

Oh my goodness.... You're not going to like this.

New disk installed. WD Purple with 256MB of cache.

2023-08-03 04:19:53.454954390  [INFO] Starting Frigate...
2023-08-03 04:19:54.425706194  [2023-08-03 04:19:54] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-08-03 04:19:54.439994444  [2023-08-03 04:19:54] peewee_migrate                 INFO    : Starting migrations
2023-08-03 04:19:54.987952903  [2023-08-03 04:19:54] peewee_migrate                 INFO    : There is nothing to migrate
2023-08-03 04:19:54.995546457  [2023-08-03 04:19:54] detector.tensorrt              INFO    : Starting detection process: 863
2023-08-03 04:19:55.009571643  [2023-08-03 04:19:54] frigate.app                    INFO    : Output process started: 865
2023-08-03 04:19:55.009639067  [2023-08-03 04:19:54] frigate.app                    INFO    : Camera processor started for frigate_street_front: 868
2023-08-03 04:19:55.009686614  [2023-08-03 04:19:54] frigate.app                    INFO    : Camera processor started for frigate_street_nw: 869
2023-08-03 04:19:55.009730176  [2023-08-03 04:19:55] frigate.app                    INFO    : Camera processor started for frigate_street_se: 870
2023-08-03 04:19:55.009771897  [2023-08-03 04:19:55] frigate.app                    INFO    : Camera processor started for frigate_backyard: 871
2023-08-03 04:19:55.009813373  [2023-08-03 04:19:55] frigate.app                    INFO    : Capture process started for frigate_street_front: 872
2023-08-03 04:19:55.009853463  [2023-08-03 04:19:55] frigate.app                    INFO    : Capture process started for frigate_street_nw: 875
2023-08-03 04:19:55.009892014  [2023-08-03 04:19:55] frigate.app                    INFO    : Capture process started for frigate_street_se: 879
2023-08-03 04:19:55.011705868  [2023-08-03 04:19:55] frigate.app                    INFO    : Capture process started for frigate_backyard: 884
2023-08-03 04:19:55.378520732  [2023-08-03 04:19:55] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 333 (MiB)
2023-08-03 04:19:55.763233541  [2023-08-03 04:19:55] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 34 MiB
2023-08-03 04:19:55.764093797  [2023-08-03 04:19:55] frigate.detectors.plugins.tensorrt WARNING : Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
2023-08-03 04:19:56.162063126  [2023-08-03 04:19:56] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +193, GPU +74, now: CPU 501, GPU 443 (MiB)
2023-08-03 04:19:56.257283421  [2023-08-03 04:19:56] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +110, GPU +46, now: CPU 611, GPU 489 (MiB)
2023-08-03 04:19:56.258464810  [2023-08-03 04:19:56] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
2023-08-03 04:19:56.261168853  [2023-08-03 04:19:56] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 577, GPU 481 (MiB)
2023-08-03 04:19:56.261219970  [2023-08-03 04:19:56] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 577, GPU 489 (MiB)
2023-08-03 04:19:56.261269244  [2023-08-03 04:19:56] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
2023-08-03 04:20:12.787904577  [2023-08-03 04:20:12] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_backyard/19.56.mp4 in 2.5221059322357178 seconds.
2023-08-03 04:20:55.122518712  [2023-08-03 04:20:55] frigate.record                 DEBUG   : Start expire recordings (new).
2023-08-03 04:20:55.122652621  [2023-08-03 04:20:55] frigate.record                 DEBUG   : Start deleted cameras.
2023-08-03 04:20:56.356670260  [2023-08-03 04:20:56] frigate.record                 DEBUG   : Expiring 0 recordings
2023-08-03 04:21:45.205604388  [2023-08-03 04:21:45] frigate.record                 DEBUG   : End deleted cameras.
2023-08-03 04:21:45.205639898  [2023-08-03 04:21:45] frigate.record                 DEBUG   : Start all cameras.
2023-08-03 04:21:45.205661157  [2023-08-03 04:21:45] frigate.record                 DEBUG   : Start camera: frigate_street_front.
2023-08-03 04:21:47.243764624  [2023-08-03 04:21:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_front/19.56.mp4 in 2.0044679641723633 seconds.
2023-08-03 04:21:49.254817989  [2023-08-03 04:21:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_nw/19.56.mp4 in 1.9544799327850342 seconds.
2023-08-03 04:21:53.919149974  [2023-08-03 04:21:53] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_se/19.56.mp4 in 4.525280952453613 seconds.
2023-08-03 04:21:54.012958821  [2023-08-03 04:21:54] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_backyard. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-08-03 04:21:54.014710659  [2023-08-03 04:21:54] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_front. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-08-03 04:21:54.016498503  [2023-08-03 04:21:54] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_nw. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-08-03 04:21:54.018275313  [2023-08-03 04:21:54] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_se. Keeping the 5 most recent segments out of 10 and discarding the rest...
2023-08-03 04:21:58.768373312  [2023-08-03 04:21:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_backyard/20.56.mp4 in 4.737589120864868 seconds.
2023-08-03 04:22:03.729708977  [2023-08-03 04:22:03] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_backyard/21.06.mp4 in 4.7328691482543945 seconds.
2023-08-03 04:22:08.625437335  [2023-08-03 04:22:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_backyard/21.16.mp4 in 4.742047071456909 seconds.
2023-08-03 04:22:13.385879126  [2023-08-03 04:22:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_backyard/21.26.mp4 in 4.679741144180298 seconds.
2023-08-03 04:22:18.209952803  [2023-08-03 04:22:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_backyard/21.36.mp4 in 4.757878065109253 seconds.
2023-08-03 04:22:23.101522186  [2023-08-03 04:22:23] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_front/20.56.mp4 in 4.820703983306885 seconds.
2023-08-03 04:22:28.631776490  [2023-08-03 04:22:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_front/21.06.mp4 in 5.455377101898193 seconds.
2023-08-03 04:22:34.302040400  [2023-08-03 04:22:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_front/21.16.mp4 in 5.612601041793823 seconds.
2023-08-03 04:22:40.335262742  [2023-08-03 04:22:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_front/21.26.mp4 in 5.985213994979858 seconds.
2023-08-03 04:22:45.833339667  [2023-08-03 04:22:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_front/21.36.mp4 in 5.4112749099731445 seconds.
2023-08-03 04:22:51.794360386  [2023-08-03 04:22:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_nw/20.56.mp4 in 5.880620956420898 seconds.
2023-08-03 04:22:58.032433328  [2023-08-03 04:22:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_nw/21.06.mp4 in 6.177794933319092 seconds.
2023-08-03 04:23:03.210026685  [2023-08-03 04:23:03] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_nw/21.16.mp4 in 5.117582082748413 seconds.
2023-08-03 04:23:09.226495471  [2023-08-03 04:23:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_nw/21.26.mp4 in 5.985745906829834 seconds.
2023-08-03 04:23:14.174410126  [2023-08-03 04:23:14] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_nw/21.36.mp4 in 4.880232810974121 seconds.
2023-08-03 04:23:19.089222670  [2023-08-03 04:23:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_se/20.56.mp4 in 4.893500804901123 seconds.
2023-08-03 04:23:24.142465128  [2023-08-03 04:23:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_se/21.06.mp4 in 5.025255918502808 seconds.
2023-08-03 04:23:29.631256754  [2023-08-03 04:23:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_se/21.16.mp4 in 5.467511892318726 seconds.
2023-08-03 04:23:34.960736849  [2023-08-03 04:23:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_se/21.26.mp4 in 5.292388916015625 seconds.
2023-08-03 04:23:40.541138545  [2023-08-03 04:23:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_se/21.36.mp4 in 5.513548851013184 seconds.
2023-08-03 04:23:40.627886659  [2023-08-03 04:23:40] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_backyard. Keeping the 5 most recent segments out of 11 and discarding the rest...
2023-08-03 04:23:40.630100367  [2023-08-03 04:23:40] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_front. Keeping the 5 most recent segments out of 11 and discarding the rest...
2023-08-03 04:23:40.632241008  [2023-08-03 04:23:40] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_nw. Keeping the 5 most recent segments out of 11 and discarding the rest...
2023-08-03 04:23:40.634418781  [2023-08-03 04:23:40] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_se. Keeping the 5 most recent segments out of 11 and discarding the rest...
2023-08-03 04:23:46.404347740  [2023-08-03 04:23:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_backyard/22.46.mp4 in 5.7559661865234375 seconds.
2023-08-03 04:23:52.343933351  [2023-08-03 04:23:52] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_backyard/22.56.mp4 in 5.874557018280029 seconds.
2023-08-03 04:23:59.456499955  [2023-08-03 04:23:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_backyard/23.06.mp4 in 7.040319919586182 seconds.
2023-08-03 04:24:05.479691059  [2023-08-03 04:24:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_backyard/23.16.mp4 in 5.956945896148682 seconds.
2023-08-03 04:24:12.540863332  [2023-08-03 04:24:12] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_backyard/23.26.mp4 in 7.020524978637695 seconds.
2023-08-03 04:24:18.811512111  [2023-08-03 04:24:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_front/22.46.mp4 in 6.18199610710144 seconds.
2023-08-03 04:24:24.805483562  [2023-08-03 04:24:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_front/22.56.mp4 in 5.932255029678345 seconds.
2023-08-03 04:24:31.142787123  [2023-08-03 04:24:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/03/frigate_street_front/23.06.mp4 in 6.307138919830322 seconds.

it's worse than it was with the 3TB WD Red 9yo drive. And this is with h265 enabled.

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Aug 3, 2023

Hmm, that doesn't really make sense why the write would be so much slower than mine with a similar drive. What are the sizes of these segments that are being written?

@Majestic7979
Copy link
Author

image

@Majestic7979
Copy link
Author

2023-08-03 14:19:32.163500059  [2023-08-03 14:19:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/19.11.mp4 in 2.3702409267425537 seconds.
2023-08-03 14:19:34.750904935  [2023-08-03 14:19:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/19.11.mp4 in 2.42980694770813 seconds.
2023-08-03 14:19:37.291660894  [2023-08-03 14:19:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/19.12.mp4 in 2.370975971221924 seconds.
2023-08-03 14:19:39.856338630  [2023-08-03 14:19:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/19.21.mp4 in 2.387183904647827 seconds.
2023-08-03 14:19:42.320618212  [2023-08-03 14:19:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/19.21.mp4 in 2.2707650661468506 seconds.
2023-08-03 14:19:44.841621678  [2023-08-03 14:19:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/19.22.mp4 in 2.3598508834838867 seconds.
2023-08-03 14:19:47.549256270  [2023-08-03 14:19:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/19.22.mp4 in 2.559983968734741 seconds.
2023-08-03 14:19:50.208716583  [2023-08-03 14:19:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/19.31.mp4 in 2.4966728687286377 seconds.
2023-08-03 14:19:52.670152103  [2023-08-03 14:19:52] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/19.31.mp4 in 2.3442399501800537 seconds.
2023-08-03 14:19:55.209041224  [2023-08-03 14:19:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/19.32.mp4 in 2.4183390140533447 seconds.
2023-08-03 14:19:57.720150962  [2023-08-03 14:19:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/19.32.mp4 in 2.3575069904327393 seconds.
2023-08-03 14:20:00.198139367  [2023-08-03 14:20:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/19.41.mp4 in 2.3297269344329834 seconds.
2023-08-03 14:20:02.698379064  [2023-08-03 14:20:02] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/19.42.mp4 in 2.3683199882507324 seconds.
2023-08-03 14:20:05.160074517  [2023-08-03 14:20:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/19.42.mp4 in 2.3257880210876465 seconds.
2023-08-03 14:20:07.664829704  [2023-08-03 14:20:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/19.42.mp4 in 2.378380060195923 seconds.
2023-08-03 14:20:10.204670323  [2023-08-03 14:20:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/19.52.mp4 in 2.368601083755493 seconds.
2023-08-03 14:20:12.682023648  [2023-08-03 14:20:12] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/19.52.mp4 in 2.331537961959839 seconds.
2023-08-03 14:20:15.193581933  [2023-08-03 14:20:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/19.52.mp4 in 2.339665174484253 seconds.
2023-08-03 14:20:17.864169304  [2023-08-03 14:20:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/19.52.mp4 in 2.449936866760254 seconds.
2023-08-03 14:20:20.769086427  [2023-08-03 14:20:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/20.02.mp4 in 2.7255070209503174 seconds.
2023-08-03 14:20:22.847587848  [2023-08-03 14:20:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/20.02.mp4 in 1.9080519676208496 seconds.
2023-08-03 14:20:24.984645479  [2023-08-03 14:20:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/20.02.mp4 in 1.9663970470428467 seconds.
2023-08-03 14:20:27.191442805  [2023-08-03 14:20:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/20.02.mp4 in 2.040337085723877 seconds.
2023-08-03 14:20:29.302775605  [2023-08-03 14:20:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/20.12.mp4 in 1.9202930927276611 seconds.
2023-08-03 14:20:31.768727848  [2023-08-03 14:20:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/20.10.mp4 in 2.3020501136779785 seconds.
2023-08-03 14:20:34.311363941  [2023-08-03 14:20:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/20.11.mp4 in 2.3632290363311768 seconds.
2023-08-03 14:20:36.822147530  [2023-08-03 14:20:36] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/20.11.mp4 in 2.338088035583496 seconds.
2023-08-03 14:20:39.351773378  [2023-08-03 14:20:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/20.20.mp4 in 2.3604941368103027 seconds.
2023-08-03 14:20:42.013727179  [2023-08-03 14:20:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/20.21.mp4 in 2.4603331089019775 seconds.
2023-08-03 14:20:44.563527592  [2023-08-03 14:20:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/20.21.mp4 in 2.367445945739746 seconds.
2023-08-03 14:20:47.097552919  [2023-08-03 14:20:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/20.21.mp4 in 2.35626482963562 seconds.
2023-08-03 14:20:49.694301046  [2023-08-03 14:20:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/20.31.mp4 in 2.431833028793335 seconds.
2023-08-03 14:20:52.314745250  [2023-08-03 14:20:52] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/20.31.mp4 in 2.4385690689086914 seconds.
2023-08-03 14:20:54.831223953  [2023-08-03 14:20:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/20.31.mp4 in 2.336587905883789 seconds.
2023-08-03 14:20:57.502352758  [2023-08-03 14:20:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/20.31.mp4 in 2.498988151550293 seconds.
2023-08-03 14:21:00.031157380  [2023-08-03 14:21:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/20.41.mp4 in 2.356374979019165 seconds.
2023-08-03 14:21:02.560189538  [2023-08-03 14:21:02] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/20.41.mp4 in 2.358285903930664 seconds.
2023-08-03 14:21:05.070511785  [2023-08-03 14:21:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/20.42.mp4 in 2.3379909992218018 seconds.
2023-08-03 14:21:07.718641721  [2023-08-03 14:21:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/20.41.mp4 in 2.4814000129699707 seconds.
2023-08-03 14:21:10.287251900  [2023-08-03 14:21:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/20.51.mp4 in 2.3890349864959717 seconds.
2023-08-03 14:21:12.882233516  [2023-08-03 14:21:12] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/20.52.mp4 in 2.4217190742492676 seconds.
2023-08-03 14:21:15.420976834  [2023-08-03 14:21:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/20.52.mp4 in 2.395045042037964 seconds.
2023-08-03 14:21:17.934856289  [2023-08-03 14:21:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/20.52.mp4 in 2.3708009719848633 seconds.
2023-08-03 14:21:20.473065992  [2023-08-03 14:21:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/21.01.mp4 in 2.3999359607696533 seconds.
2023-08-03 14:21:23.012975774  [2023-08-03 14:21:23] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/21.02.mp4 in 2.3540878295898438 seconds.
2023-08-03 14:21:25.101773157  [2023-08-03 14:21:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/21.02.mp4 in 1.9438328742980957 seconds.
2023-08-03 14:21:27.535424821  [2023-08-03 14:21:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/21.02.mp4 in 2.3012290000915527 seconds.
2023-08-03 14:21:29.994488776  [2023-08-03 14:21:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/21.12.mp4 in 2.319749116897583 seconds.
2023-08-03 14:21:32.109597233  [2023-08-03 14:21:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/21.12.mp4 in 1.9584770202636719 seconds.
2023-08-03 14:21:34.724794118  [2023-08-03 14:21:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/21.10.mp4 in 2.4499900341033936 seconds.
2023-08-03 14:21:36.780068455  [2023-08-03 14:21:36] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/21.13.mp4 in 1.9249341487884521 seconds.
2023-08-03 14:21:39.335855776  [2023-08-03 14:21:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/21.22.mp4 in 2.3884830474853516 seconds.
2023-08-03 14:21:41.867399172  [2023-08-03 14:21:41] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/21.21.mp4 in 2.3855910301208496 seconds.
2023-08-03 14:21:44.413669205  [2023-08-03 14:21:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/21.21.mp4 in 2.405496835708618 seconds.
2023-08-03 14:21:46.913802036  [2023-08-03 14:21:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/21.20.mp4 in 2.3566720485687256 seconds.
2023-08-03 14:21:49.014668534  [2023-08-03 14:21:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/21.32.mp4 in 1.9589760303497314 seconds.
2023-08-03 14:21:51.682793105  [2023-08-03 14:21:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/21.31.mp4 in 2.4686758518218994 seconds.
2023-08-03 14:21:54.245844687  [2023-08-03 14:21:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/21.31.mp4 in 2.398516893386841 seconds.
2023-08-03 14:21:56.896743296  [2023-08-03 14:21:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/21.31.mp4 in 2.4035580158233643 seconds.
2023-08-03 14:21:59.487602652  [2023-08-03 14:21:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/21.41.mp4 in 2.4045090675354004 seconds.
2023-08-03 14:22:02.285682784  [2023-08-03 14:22:02] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/21.41.mp4 in 2.6407501697540283 seconds.
2023-08-03 14:22:04.830794708  [2023-08-03 14:22:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/21.41.mp4 in 2.3463850021362305 seconds.
2023-08-03 14:22:07.418376739  [2023-08-03 14:22:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/21.41.mp4 in 2.452025890350342 seconds.
2023-08-03 14:22:09.928626053  [2023-08-03 14:22:09] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/21.51.mp4 in 2.3080480098724365 seconds.
2023-08-03 14:22:12.450029547  [2023-08-03 14:22:12] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/21.51.mp4 in 2.3462109565734863 seconds.
2023-08-03 14:22:15.007684824  [2023-08-03 14:22:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/21.52.mp4 in 2.374782085418701 seconds.
2023-08-03 14:22:17.510535677  [2023-08-03 14:22:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/21.51.mp4 in 2.337394952774048 seconds.
2023-08-03 14:22:20.196619594  [2023-08-03 14:22:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/22.01.mp4 in 2.491093873977661 seconds.
2023-08-03 14:22:22.765812303  [2023-08-03 14:22:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/22.02.mp4 in 2.397649049758911 seconds.
2023-08-03 14:22:25.367364503  [2023-08-03 14:22:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/22.02.mp4 in 2.386660099029541 seconds.
2023-08-03 14:22:27.986488342  [2023-08-03 14:22:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/22.02.mp4 in 2.429266929626465 seconds.
2023-08-03 14:22:30.546443358  [2023-08-03 14:22:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/22.12.mp4 in 2.354919910430908 seconds.
2023-08-03 14:22:33.073565325  [2023-08-03 14:22:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/22.12.mp4 in 2.3511710166931152 seconds.
2023-08-03 14:22:35.229719848  [2023-08-03 14:22:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/22.12.mp4 in 1.96048903465271 seconds.
2023-08-03 14:22:38.053267763  [2023-08-03 14:22:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/22.12.mp4 in 2.646636962890625 seconds.
2023-08-03 14:22:40.638155392  [2023-08-03 14:22:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/22.22.mp4 in 2.375730037689209 seconds.
2023-08-03 14:22:42.732387357  [2023-08-03 14:22:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/22.22.mp4 in 1.9324359893798828 seconds.
2023-08-03 14:22:45.251746426  [2023-08-03 14:22:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/22.21.mp4 in 2.353137969970703 seconds.
2023-08-03 14:22:47.719502368  [2023-08-03 14:22:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/22.22.mp4 in 2.329284906387329 seconds.
2023-08-03 14:22:49.817262864  [2023-08-03 14:22:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/22.32.mp4 in 1.9554119110107422 seconds.
2023-08-03 14:22:52.696477290  [2023-08-03 14:22:52] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/22.30.mp4 in 2.715676784515381 seconds.
2023-08-03 14:22:55.278286235  [2023-08-03 14:22:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/22.31.mp4 in 2.4396400451660156 seconds.
2023-08-03 14:22:57.483339122  [2023-08-03 14:22:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/22.32.mp4 in 2.0706288814544678 seconds.
2023-08-03 14:22:59.973747022  [2023-08-03 14:22:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/22.40.mp4 in 2.333799123764038 seconds.
2023-08-03 14:23:02.549146541  [2023-08-03 14:23:02] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/22.41.mp4 in 2.4223108291625977 seconds.
2023-08-03 14:23:05.064544496  [2023-08-03 14:23:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/22.41.mp4 in 2.3736190795898438 seconds.
2023-08-03 14:23:07.562082884  [2023-08-03 14:23:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/22.41.mp4 in 2.3164749145507812 seconds.
2023-08-03 14:23:10.062200772  [2023-08-03 14:23:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/22.51.mp4 in 2.342460870742798 seconds.
2023-08-03 14:23:12.517533958  [2023-08-03 14:23:12] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/22.51.mp4 in 2.3266689777374268 seconds.
2023-08-03 14:23:15.001941717  [2023-08-03 14:23:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/22.52.mp4 in 2.3481528759002686 seconds.
2023-08-03 14:23:17.474583151  [2023-08-03 14:23:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/22.51.mp4 in 2.3410351276397705 seconds.
2023-08-03 14:23:19.975976762  [2023-08-03 14:23:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/23.01.mp4 in 2.3696539402008057 seconds.
2023-08-03 14:23:22.632387927  [2023-08-03 14:23:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/23.01.mp4 in 2.4524309635162354 seconds.
2023-08-03 14:23:25.187923332  [2023-08-03 14:23:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/23.01.mp4 in 2.380047082901001 seconds.
2023-08-03 14:23:28.209198357  [2023-08-03 14:23:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/23.01.mp4 in 2.8337738513946533 seconds.
2023-08-03 14:23:30.745951596  [2023-08-03 14:23:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/23.11.mp4 in 2.3607330322265625 seconds.
2023-08-03 14:23:33.411722173  [2023-08-03 14:23:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/23.11.mp4 in 2.4681410789489746 seconds.
2023-08-03 14:23:35.909426482  [2023-08-03 14:23:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/23.12.mp4 in 2.325446844100952 seconds.
2023-08-03 14:23:38.422196513  [2023-08-03 14:23:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/23.11.mp4 in 2.37833309173584 seconds.
2023-08-03 14:23:40.988762121  [2023-08-03 14:23:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/23.22.mp4 in 2.286691904067993 seconds.
2023-08-03 14:23:43.533356086  [2023-08-03 14:23:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/23.21.mp4 in 2.353008985519409 seconds.
2023-08-03 14:23:46.247479507  [2023-08-03 14:23:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/23.22.mp4 in 2.5447609424591064 seconds.
2023-08-03 14:23:48.878111552  [2023-08-03 14:23:48] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/23.21.mp4 in 2.437314987182617 seconds.
2023-08-03 14:23:51.022409856  [2023-08-03 14:23:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/23.32.mp4 in 1.953768014907837 seconds.
2023-08-03 14:23:53.715397592  [2023-08-03 14:23:53] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/23.32.mp4 in 2.508845806121826 seconds.
2023-08-03 14:23:56.516640283  [2023-08-03 14:23:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/23.32.mp4 in 2.6636080741882324 seconds.
2023-08-03 14:23:59.000043865  [2023-08-03 14:23:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/23.31.mp4 in 2.3062100410461426 seconds.
2023-08-03 14:24:01.576061587  [2023-08-03 14:24:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/23.40.mp4 in 2.3661811351776123 seconds.
2023-08-03 14:24:04.099334738  [2023-08-03 14:24:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/23.42.mp4 in 2.3293230533599854 seconds.
2023-08-03 14:24:06.155093090  [2023-08-03 14:24:06] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/23.42.mp4 in 1.9048378467559814 seconds.
2023-08-03 14:24:08.724743604  [2023-08-03 14:24:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/23.41.mp4 in 2.4268898963928223 seconds.
2023-08-03 14:24:11.223090978  [2023-08-03 14:24:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/23.51.mp4 in 2.305938959121704 seconds.
2023-08-03 14:24:13.312519742  [2023-08-03 14:24:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/23.52.mp4 in 1.9328010082244873 seconds.
2023-08-03 14:24:15.889792930  [2023-08-03 14:24:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/23.51.mp4 in 2.4014041423797607 seconds.
2023-08-03 14:24:18.391493537  [2023-08-03 14:24:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/23.52.mp4 in 2.3746209144592285 seconds.
2023-08-03 14:24:20.897682459  [2023-08-03 14:24:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/24.01.mp4 in 2.3260440826416016 seconds.
2023-08-03 14:24:23.499121060  [2023-08-03 14:24:23] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/24.01.mp4 in 2.47251296043396 seconds.
2023-08-03 14:24:26.031210294  [2023-08-03 14:24:26] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/24.01.mp4 in 2.3910951614379883 seconds.
2023-08-03 14:24:28.521454792  [2023-08-03 14:24:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/24.02.mp4 in 2.358487129211426 seconds.
2023-08-03 14:24:31.031556257  [2023-08-03 14:24:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/24.11.mp4 in 2.376102924346924 seconds.
2023-08-03 14:24:33.580630499  [2023-08-03 14:24:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/24.11.mp4 in 2.3732810020446777 seconds.
2023-08-03 14:24:36.019334374  [2023-08-03 14:24:36] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/24.11.mp4 in 2.2885661125183105 seconds.
2023-08-03 14:24:38.196799372  [2023-08-03 14:24:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/24.12.mp4 in 2.0257930755615234 seconds.
2023-08-03 14:24:41.019673349  [2023-08-03 14:24:41] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/24.21.mp4 in 2.675518035888672 seconds.
2023-08-03 14:24:43.398055361  [2023-08-03 14:24:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/24.21.mp4 in 2.2674639225006104 seconds.
2023-08-03 14:24:45.899370797  [2023-08-03 14:24:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/24.21.mp4 in 2.3294310569763184 seconds.
2023-08-03 14:24:48.331460688  [2023-08-03 14:24:48] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/24.20.mp4 in 2.308403968811035 seconds.
2023-08-03 14:24:50.929034148  [2023-08-03 14:24:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/24.32.mp4 in 2.463639974594116 seconds.
2023-08-03 14:24:53.469936148  [2023-08-03 14:24:53] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/24.31.mp4 in 2.372380018234253 seconds.
2023-08-03 14:24:56.028840386  [2023-08-03 14:24:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/24.31.mp4 in 2.383219003677368 seconds.
2023-08-03 14:24:58.603213700  [2023-08-03 14:24:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/24.31.mp4 in 2.404780864715576 seconds.
2023-08-03 14:25:00.756426765  [2023-08-03 14:25:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/24.42.mp4 in 1.9271361827850342 seconds.
2023-08-03 14:25:03.267997963  [2023-08-03 14:25:03] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/24.41.mp4 in 2.3346009254455566 seconds.
2023-08-03 14:25:06.033740630  [2023-08-03 14:25:06] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/24.41.mp4 in 2.5607290267944336 seconds.
2023-08-03 14:25:08.607884860  [2023-08-03 14:25:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/24.41.mp4 in 2.407052993774414 seconds.
2023-08-03 14:25:11.135850459  [2023-08-03 14:25:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/24.50.mp4 in 2.3527917861938477 seconds.
2023-08-03 14:25:13.669245455  [2023-08-03 14:25:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/24.51.mp4 in 2.350583076477051 seconds.
2023-08-03 14:25:16.514840329  [2023-08-03 14:25:16] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/24.52.mp4 in 2.685544013977051 seconds.
2023-08-03 14:25:19.057914576  [2023-08-03 14:25:19] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/24.51.mp4 in 2.3709030151367188 seconds.
2023-08-03 14:25:21.579239428  [2023-08-03 14:25:21] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/25.01.mp4 in 2.3375871181488037 seconds.
2023-08-03 14:25:24.140172506  [2023-08-03 14:25:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/25.02.mp4 in 2.3885080814361572 seconds.
2023-08-03 14:25:26.735904587  [2023-08-03 14:25:26] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/25.02.mp4 in 2.3933138847351074 seconds.
2023-08-03 14:25:29.313920337  [2023-08-03 14:25:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/25.01.mp4 in 2.3993489742279053 seconds.
2023-08-03 14:25:31.881420803  [2023-08-03 14:25:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/25.11.mp4 in 2.379357099533081 seconds.
2023-08-03 14:25:33.990483523  [2023-08-03 14:25:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/25.12.mp4 in 1.943528175354004 seconds.
2023-08-03 14:25:36.056498710  [2023-08-03 14:25:36] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/25.12.mp4 in 1.9061250686645508 seconds.
2023-08-03 14:25:38.646850679  [2023-08-03 14:25:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/25.12.mp4 in 2.4039080142974854 seconds.
2023-08-03 14:25:41.200109607  [2023-08-03 14:25:41] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/25.21.mp4 in 2.3789079189300537 seconds.
2023-08-03 14:25:44.014619682  [2023-08-03 14:25:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/25.20.mp4 in 2.6347670555114746 seconds.
2023-08-03 14:25:46.581746439  [2023-08-03 14:25:46] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/25.21.mp4 in 2.3917739391326904 seconds.
2023-08-03 14:25:49.099040367  [2023-08-03 14:25:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/25.22.mp4 in 2.3855979442596436 seconds.
2023-08-03 14:25:51.587403690  [2023-08-03 14:25:51] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/25.32.mp4 in 2.334757089614868 seconds.
2023-08-03 14:25:54.065043962  [2023-08-03 14:25:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/25.31.mp4 in 2.3455810546875 seconds.
2023-08-03 14:25:56.616792167  [2023-08-03 14:25:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/25.31.mp4 in 2.42366099357605 seconds.
2023-08-03 14:25:58.709040952  [2023-08-03 14:25:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/25.32.mp4 in 1.947484016418457 seconds.
2023-08-03 14:26:00.754209614  [2023-08-03 14:26:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/25.42.mp4 in 1.9031150341033936 seconds.
2023-08-03 14:26:03.253392215  [2023-08-03 14:26:03] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/25.41.mp4 in 2.365960121154785 seconds.
2023-08-03 14:26:05.767223125  [2023-08-03 14:26:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/25.41.mp4 in 2.365401029586792 seconds.
2023-08-03 14:26:08.255899501  [2023-08-03 14:26:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/25.40.mp4 in 2.3513710498809814 seconds.
2023-08-03 14:26:10.782063506  [2023-08-03 14:26:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/25.50.mp4 in 2.384423017501831 seconds.
2023-08-03 14:26:13.310525464  [2023-08-03 14:26:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/25.51.mp4 in 2.391251802444458 seconds.
2023-08-03 14:26:15.754117032  [2023-08-03 14:26:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/25.51.mp4 in 2.3254621028900146 seconds.
2023-08-03 14:26:18.266099541  [2023-08-03 14:26:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/25.50.mp4 in 2.371845006942749 seconds.
2023-08-03 14:26:20.837687478  [2023-08-03 14:26:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/26.01.mp4 in 2.4101738929748535 seconds.
2023-08-03 14:26:23.373827592  [2023-08-03 14:26:23] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/26.01.mp4 in 2.3461129665374756 seconds.
2023-08-03 14:26:26.037354562  [2023-08-03 14:26:26] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/26.01.mp4 in 2.448881149291992 seconds.
2023-08-03 14:26:29.328000772  [2023-08-03 14:26:29] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/26.01.mp4 in 3.127526044845581 seconds.
2023-08-03 14:26:31.818532152  [2023-08-03 14:26:31] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/26.11.mp4 in 2.3243517875671387 seconds.
2023-08-03 14:26:34.397077316  [2023-08-03 14:26:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/26.12.mp4 in 2.3802108764648438 seconds.
2023-08-03 14:26:36.972077715  [2023-08-03 14:26:36] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/26.12.mp4 in 2.416543960571289 seconds.
2023-08-03 14:26:39.452446156  [2023-08-03 14:26:39] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/26.11.mp4 in 2.344672203063965 seconds.
2023-08-03 14:26:42.023600329  [2023-08-03 14:26:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/26.21.mp4 in 2.383319854736328 seconds.
2023-08-03 14:26:44.625540441  [2023-08-03 14:26:44] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/26.22.mp4 in 2.4179909229278564 seconds.
2023-08-03 14:26:47.255196905  [2023-08-03 14:26:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/26.22.mp4 in 2.366335868835449 seconds.
2023-08-03 14:26:49.819946205  [2023-08-03 14:26:49] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/26.21.mp4 in 2.303428888320923 seconds.
2023-08-03 14:26:52.918859719  [2023-08-03 14:26:52] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/26.32.mp4 in 2.9027810096740723 seconds.
2023-08-03 14:26:55.439179201  [2023-08-03 14:26:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/26.32.mp4 in 2.339845895767212 seconds.
2023-08-03 14:26:57.638640341  [2023-08-03 14:26:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/26.32.mp4 in 2.009586811065674 seconds.
2023-08-03 14:27:00.263843687  [2023-08-03 14:27:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/26.31.mp4 in 2.4660890102386475 seconds.
2023-08-03 14:27:02.838318978  [2023-08-03 14:27:02] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/26.42.mp4 in 2.3936328887939453 seconds.
2023-08-03 14:27:05.013842967  [2023-08-03 14:27:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/26.42.mp4 in 1.970905065536499 seconds.
2023-08-03 14:27:07.560297156  [2023-08-03 14:27:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/26.41.mp4 in 2.3664040565490723 seconds.
2023-08-03 14:27:10.549960005  [2023-08-03 14:27:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/26.42.mp4 in 2.8108599185943604 seconds.
2023-08-03 14:27:13.025654896  [2023-08-03 14:27:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/26.52.mp4 in 2.3419721126556396 seconds.
2023-08-03 14:27:15.538724611  [2023-08-03 14:27:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/26.51.mp4 in 2.3452529907226562 seconds.
2023-08-03 14:27:18.001435095  [2023-08-03 14:27:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/26.51.mp4 in 2.3156239986419678 seconds.
2023-08-03 14:27:20.532145709  [2023-08-03 14:27:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/26.52.mp4 in 2.3866240978240967 seconds.
2023-08-03 14:27:22.559372203  [2023-08-03 14:27:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/27.02.mp4 in 1.8843159675598145 seconds.
2023-08-03 14:27:25.100749104  [2023-08-03 14:27:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/27.01.mp4 in 2.3894729614257812 seconds.
2023-08-03 14:27:27.694167945  [2023-08-03 14:27:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/27.01.mp4 in 2.4756410121917725 seconds.
2023-08-03 14:27:30.499057313  [2023-08-03 14:27:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/27.02.mp4 in 2.683281183242798 seconds.
2023-08-03 14:27:33.022899825  [2023-08-03 14:27:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/27.10.mp4 in 2.3867900371551514 seconds.
2023-08-03 14:27:35.501781549  [2023-08-03 14:27:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/27.11.mp4 in 2.350109100341797 seconds.
2023-08-03 14:27:37.943888347  [2023-08-03 14:27:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/27.11.mp4 in 2.3192670345306396 seconds.
2023-08-03 14:27:40.027799087  [2023-08-03 14:27:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/27.12.mp4 in 1.9375410079956055 seconds.
2023-08-03 14:27:42.512809532  [2023-08-03 14:27:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/27.21.mp4 in 2.3315951824188232 seconds.
2023-08-03 14:27:45.010960475  [2023-08-03 14:27:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/27.21.mp4 in 2.357573986053467 seconds.
2023-08-03 14:27:47.615870139  [2023-08-03 14:27:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/27.21.mp4 in 2.4578561782836914 seconds.
2023-08-03 14:27:50.583951750  [2023-08-03 14:27:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/27.20.mp4 in 2.3993918895721436 seconds.
2023-08-03 14:27:53.158514940  [2023-08-03 14:27:53] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/27.31.mp4 in 2.3878109455108643 seconds.
2023-08-03 14:27:55.725439336  [2023-08-03 14:27:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/27.31.mp4 in 2.398714065551758 seconds.
2023-08-03 14:27:58.420780859  [2023-08-03 14:27:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/27.32.mp4 in 2.519728899002075 seconds.
2023-08-03 14:28:00.941776076  [2023-08-03 14:28:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/27.31.mp4 in 2.3632309436798096 seconds.
2023-08-03 14:28:03.454155869  [2023-08-03 14:28:03] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/27.41.mp4 in 2.323745012283325 seconds.
2023-08-03 14:28:05.962604571  [2023-08-03 14:28:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/27.42.mp4 in 2.3213109970092773 seconds.
2023-08-03 14:28:08.506945196  [2023-08-03 14:28:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/27.42.mp4 in 2.3993871212005615 seconds.
2023-08-03 14:28:11.060003023  [2023-08-03 14:28:11] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/27.41.mp4 in 2.387296199798584 seconds.
2023-08-03 14:28:13.609159701  [2023-08-03 14:28:13] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/27.52.mp4 in 2.3943231105804443 seconds.
2023-08-03 14:28:15.788360253  [2023-08-03 14:28:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/28.02.mp4 in 1.982496976852417 seconds.
2023-08-03 14:28:18.359837670  [2023-08-03 14:28:18] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/27.52.mp4 in 2.375046968460083 seconds.
2023-08-03 14:28:22.242764795  [2023-08-03 14:28:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/27.52.mp4 in 3.243664026260376 seconds.
2023-08-03 14:28:24.403410247  [2023-08-03 14:28:24] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/28.02.mp4 in 1.9887919425964355 seconds.
2023-08-03 14:28:27.387892101  [2023-08-03 14:28:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/27.51.mp4 in 2.771649122238159 seconds.
2023-08-03 14:28:30.061150641  [2023-08-03 14:28:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/28.10.mp4 in 2.4549992084503174 seconds.
2023-08-03 14:28:32.184970135  [2023-08-03 14:28:32] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/28.02.mp4 in 1.952970027923584 seconds.
2023-08-03 14:28:34.759961422  [2023-08-03 14:28:34] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/28.11.mp4 in 2.401607036590576 seconds.
2023-08-03 14:28:37.469302190  [2023-08-03 14:28:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/28.10.mp4 in 2.562166929244995 seconds.
2023-08-03 14:28:40.047645330  [2023-08-03 14:28:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/28.01.mp4 in 2.413717031478882 seconds.
2023-08-03 14:28:42.638427131  [2023-08-03 14:28:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/28.12.mp4 in 2.422173023223877 seconds.
2023-08-03 14:28:45.258413414  [2023-08-03 14:28:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/28.21.mp4 in 2.4099559783935547 seconds.
2023-08-03 14:28:47.751021931  [2023-08-03 14:28:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/28.31.mp4 in 2.339099884033203 seconds.
2023-08-03 14:28:50.262203971  [2023-08-03 14:28:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/28.21.mp4 in 2.369986057281494 seconds.
2023-08-03 14:28:53.055842953  [2023-08-03 14:28:53] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/28.31.mp4 in 2.6251370906829834 seconds.
2023-08-03 14:28:55.492627196  [2023-08-03 14:28:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/28.21.mp4 in 2.3047220706939697 seconds.
2023-08-03 14:28:57.985834078  [2023-08-03 14:28:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/28.31.mp4 in 2.361255168914795 seconds.
2023-08-03 14:29:00.567902150  [2023-08-03 14:29:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/28.22.mp4 in 2.4547119140625 seconds.
2023-08-03 14:29:03.111619914  [2023-08-03 14:29:03] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/28.41.mp4 in 2.3828392028808594 seconds.
2023-08-03 14:29:05.690494518  [2023-08-03 14:29:05] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/28.41.mp4 in 2.40447998046875 seconds.
2023-08-03 14:29:08.292727014  [2023-08-03 14:29:08] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/28.41.mp4 in 2.471903085708618 seconds.
2023-08-03 14:29:10.757082182  [2023-08-03 14:29:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/28.32.mp4 in 2.331946849822998 seconds.
2023-08-03 14:29:12.822985996  [2023-08-03 14:29:12] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/28.42.mp4 in 1.9374151229858398 seconds.
2023-08-03 14:29:15.325341389  [2023-08-03 14:29:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/28.52.mp4 in 2.3583688735961914 seconds.
2023-08-03 14:29:17.791907052  [2023-08-03 14:29:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/29.02.mp4 in 2.3263511657714844 seconds.
2023-08-03 14:29:20.433081022  [2023-08-03 14:29:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/28.52.mp4 in 2.355401039123535 seconds.
2023-08-03 14:29:22.938488319  [2023-08-03 14:29:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/29.02.mp4 in 1.9529900550842285 seconds.
2023-08-03 14:29:25.444794388  [2023-08-03 14:29:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/28.52.mp4 in 2.338620901107788 seconds.
2023-08-03 14:29:28.005913086  [2023-08-03 14:29:28] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/29.02.mp4 in 2.396759033203125 seconds.
2023-08-03 14:29:30.705182626  [2023-08-03 14:29:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/28.51.mp4 in 2.533405065536499 seconds.
2023-08-03 14:29:33.288565427  [2023-08-03 14:29:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/29.01.mp4 in 2.390876054763794 seconds.
2023-08-03 14:29:35.401945834  [2023-08-03 14:29:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/29.12.mp4 in 1.9739139080047607 seconds.
2023-08-03 14:29:37.931926255  [2023-08-03 14:29:37] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/29.21.mp4 in 2.394908905029297 seconds.
2023-08-03 14:29:40.529218785  [2023-08-03 14:29:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/29.10.mp4 in 2.4172658920288086 seconds.
2023-08-03 14:29:42.996861164  [2023-08-03 14:29:42] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/29.21.mp4 in 2.3096320629119873 seconds.
2023-08-03 14:29:45.552330444  [2023-08-03 14:29:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/29.12.mp4 in 2.3552191257476807 seconds.
2023-08-03 14:29:47.696237392  [2023-08-03 14:29:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/29.22.mp4 in 1.966425895690918 seconds.
2023-08-03 14:29:50.193716881  [2023-08-03 14:29:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/29.11.mp4 in 2.3283040523529053 seconds.
2023-08-03 14:29:52.649814241  [2023-08-03 14:29:52] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/29.21.mp4 in 2.2592968940734863 seconds.
2023-08-03 14:29:55.627519891  [2023-08-03 14:29:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/29.31.mp4 in 2.3771378993988037 seconds.
2023-08-03 14:29:58.216521720  [2023-08-03 14:29:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/29.41.mp4 in 2.3978350162506104 seconds.
2023-08-03 14:30:01.970534091  [2023-08-03 14:30:01] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/29.31.mp4 in 3.573862075805664 seconds.
2023-08-03 14:30:04.994129846  [2023-08-03 14:30:04] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/29.41.mp4 in 2.856235980987549 seconds.
2023-08-03 14:30:07.583108889  [2023-08-03 14:30:07] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/29.30.mp4 in 2.3904340267181396 seconds.
2023-08-03 14:30:10.133975806  [2023-08-03 14:30:10] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/29.41.mp4 in 2.340822219848633 seconds.
2023-08-03 14:30:12.683058804  [2023-08-03 14:30:12] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/29.31.mp4 in 2.3605148792266846 seconds.
2023-08-03 14:30:15.213929874  [2023-08-03 14:30:15] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/29.42.mp4 in 2.356339931488037 seconds.
2023-08-03 14:30:17.737401755  [2023-08-03 14:30:17] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/29.51.mp4 in 2.3257079124450684 seconds.
2023-08-03 14:30:20.171445961  [2023-08-03 14:30:20] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/30.02.mp4 in 2.293044090270996 seconds.
2023-08-03 14:30:22.746278538  [2023-08-03 14:30:22] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/29.51.mp4 in 2.4144580364227295 seconds.
2023-08-03 14:30:25.275177112  [2023-08-03 14:30:25] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/30.02.mp4 in 2.3800089359283447 seconds.
2023-08-03 14:30:27.890671311  [2023-08-03 14:30:27] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/29.51.mp4 in 2.4290599822998047 seconds.
2023-08-03 14:30:30.430630429  [2023-08-03 14:30:30] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/30.01.mp4 in 2.403865098953247 seconds.
2023-08-03 14:30:33.063666605  [2023-08-03 14:30:33] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/29.52.mp4 in 2.455662965774536 seconds.
2023-08-03 14:30:35.602606205  [2023-08-03 14:30:35] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/30.02.mp4 in 2.377462863922119 seconds.
2023-08-03 14:30:38.120417456  [2023-08-03 14:30:38] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/30.12.mp4 in 2.3401949405670166 seconds.
2023-08-03 14:30:40.580307565  [2023-08-03 14:30:40] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_backyard/30.22.mp4 in 2.3199169635772705 seconds.
2023-08-03 14:30:43.068973676  [2023-08-03 14:30:43] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/30.12.mp4 in 2.3429129123687744 seconds.
2023-08-03 14:30:45.168315952  [2023-08-03 14:30:45] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_front/30.22.mp4 in 1.939743995666504 seconds.
2023-08-03 14:30:47.891899826  [2023-08-03 14:30:47] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/30.12.mp4 in 2.5829010009765625 seconds.
2023-08-03 14:30:50.419668092  [2023-08-03 14:30:50] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_nw/30.22.mp4 in 2.3739190101623535 seconds.
2023-08-03 14:30:53.253557594  [2023-08-03 14:30:53] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/30.12.mp4 in 2.647918939590454 seconds.
2023-08-03 14:30:55.805031218  [2023-08-03 14:30:55] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/13/frigate_street_se/30.21.mp4 in 2.375213146209717 seconds.

As you can see between the post and now, the time has gone down. So there is something happening process-wise we need to find out and eliminate. I don't know what it could be, but this is not acceptable, if it's the OS we need to see what's happening and see if we can tweak or turn on/off some setting. If it's Frigate we need to fix it in Frigate. But what we cannot have is reduced reliability. Regardless of the source Frigate will be seen as the make-or-break application if users install it and notice that footage is skipping... it could drive them away. I know it's more complicated than that but I'd argue a lot of people won't know and won't care to investigate. I hope we can continue through this very long issue and find the root and find a fix for it.

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Aug 3, 2023

We have already identified and fixed the processing side of recording maintenance in Frigate for 0.13, which really only comes in to play when the disk transfers are fast but frigate still falls behind.

The Copied in x seconds is only the time it took for the external call so the only reason that would be slow is due to a host or hardware issue.

@Majestic7979
Copy link
Author

Majestic7979 commented Aug 3, 2023

We have already identified and fixed the processing side of recording maintenance in Frigate for 0.13. The Copied in x seconds is only the time it took for the external call so the only reason that would be slow is due to a host or hardware issue.

Can you tell me exact what the issue is then? Because I don't think it's that simple. I bought a brand new HDD like you asked me to do. One with even more cache than yours. And my system is no slouch. Do you have a GTX1080 in your system? Do you have a SSD running Debian 12 on it capable of over 5000MB/s reads and writer? Have you got a 13th Gen Intel Processor running frigate? Fast DDR5 Mem, a whole 32GB of it? Please make a comparison with your own system and lets stop the hardware issue line because it ain't it. I am more inclined to think it's software-related.

I will draw the line on being a hardware issue now. It's not and I won't spend money on it anymore. We should have debug methods available to find out what's the issue. If Frigate is lacking a more detailed debugging interface now is the time to implement that... Please discuss with Blake and let me know how we should process but any suggestion that it's hardware I won't be receptive to it. You said that you use UNRAID with some unorthodox mixed cache situation - most people don't have that. Most people have hard drives. I bought a hard drive SPECIFICALLY marketed and optimized for NVRs.

While I attempt to be polite 100% of the time I do not attempt to be "liked", I know you won't like me for saying this but it's the truth: if Frigate wants to be taken seriously as a NVR solution it needs to work around user's limitations not the other way around. My system is orders of magnitude more than most systems that your application will ever touch - I KNOW THAT.

I know that because I read people's posts with their specs and they use far slower machines. It was an expensive machine to build for my purposes, I overprovisioned it so there would be no doubt that it's NOT THE HARDWARE. Frigate needs to consider as a project what is the average device that it will be running on and adapt. It needs to have a robust set of debug interfaces and it needs to more than anything focus on reliability because faulty footage is as good as nothing in criminal proceedings.

Understand that my comment is NOT personal - it's targeted at the application.

Understand that my comment has GOOD INTENTIONS - I want to see this app be the one thing anyone things of when they think smart NVR. And the way to get there is to make its main function work well. All the detections in the world won't matter if the file is not saved in a way that is usable in court!

The Copied in x seconds is only the time it took for the external call so the only reason that would be slow is due to a host or hardware issue.

We need to investigate this further. I don't know where to start.

system-monitor

@Majestic7979
Copy link
Author

Ps I have dyslexia and I just woke up, there will be typos.

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Aug 3, 2023

I think my comment was ambiguous, by hardware issue I was not suggesting the system in general is slow. Users run on intel nucs and don't have this problem that you are seeing. By hardware I meant something with the interface for the HDD, or a host issue by which I meant an issue with the host os / underlying configuration.

You said that you use UNRAID with some unorthodox mixed cache situation - most people don't have that. Most people have hard drives. I bought a hard drive SPECIFICALLY marketed and optimized for NVRs.

A write cache is not unorthodox, it is a feature built in to ZFS, fuse.fs in UnraidOS case, and a feature for other file systems as well. But in any case, I did tests above without using the cache and write speeds directly to the HDD were still much faster than what you are seeing for a 10MB file.

@Majestic7979
Copy link
Author

I think my comment was ambiguous, by hardware issue I was not suggesting the system in general is slow. Users run on intel nucs and don't have this problem that you are seeing. By hardware I meant something with the interface for the HDD, or a host issue by which I meant an issue with the host os / underlying configuration.

You said that you use UNRAID with some unorthodox mixed cache situation - most people don't have that. Most people have hard drives. I bought a hard drive SPECIFICALLY marketed and optimized for NVRs.

A write cache is not unorthodox, it is a feature built in to ZFS, fuse.fs in UnraidOS case, and a feature for other file systems as well. But in any case, I did tests above without using the cache and write speeds directly to the HDD were still much faster than what you are seeing for a 10MB file.

Exactly - and how many people you know are using ZFS or paying for UNRAID? My system is barebones Debian. It's much more likely to be that way. Nobody is going to be running ZFS on a Raspberry Pi for instance.

Let's stop talking about mine vs yours, it's pointless as you've just confirmed after testing, and let's focus on finding the issue - while I might be the only one, we need to be prepared for the possibility that actually, there might be other people with this issue who are not aware or have not bothered to report it.

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Aug 3, 2023

Let's stop talking about mine vs yours, it's pointless as you've just confirmed after testing

I disagree, your HDD has more cache but is otherwise similar to mine and mine has much faster write speeds for a similar mp4 file. That is useful information.

What file system do you have on the HDD? Mine uses xfs.

@Majestic7979
Copy link
Author

Let's stop talking about mine vs yours, it's pointless as you've just confirmed after testing

I disagree, your HDD has more cache but is otherwise similar to mine and mine has much faster write speeds for a similar mp4 file. That is useful information.

What file system do you have on the HDD? Mine uses xfs.

Well you're only confirming what I said - that it should not be happening because I got the HDD you told me you were using. Only more cache.

I use Ext4 which is optimized for small random operations.

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Aug 3, 2023

The physical hardware is the same, but there is a large difference in the amount of time to write a 10MB mp4 file. In my opinion this points to an issue outside of Frigate, and support here or Frigate itself does not know the details of the system on the host so it is very difficult to make specific suggestions on what to change.

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Aug 3, 2023

I have an idea for something to try, I downloaded this 12 MB sample mp4 from https://download.samplelib.com/mp4/sample-15s.mp4

From within Frigate shell I ran the ffmpeg command to optimize the metadata and move it to my HDD (no cache involved) and the rtime is 0.028 seconds

Running the test on the same file should help further reduce variables in the comparison.

ffmpeg -benchmark -hide_banner -y -i /media/frigate/exports/sample-15s.mp4 -c copy -movflags +faststart /sandbox/test.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/media/frigate/exports/sample-15s.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.43.100
  Duration: 00:00:15.65, start: 0.000000, bitrate: 6091 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 6021 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
    Metadata:
      handler_name    : ISO Media file produced by Google Inc. Created on: 08/17/2020.
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      handler_name    : ISO Media file produced by Google Inc. Created on: 08/17/2020.
      vendor_id       : [0][0][0][0]
Output #0, mp4, to '/sandbox/test.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf59.27.100
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 6021 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
    Metadata:
      handler_name    : ISO Media file produced by Google Inc. Created on: 08/17/2020.
      vendor_id       : [0][0][0][0]
  Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      handler_name    : ISO Media file produced by Google Inc. Created on: 08/17/2020.
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x5628919c1980] Starting second pass: moving the moov atom to the beginning of the file
frame=  464 fps=0.0 q=-1.0 Lsize=   11637kB time=00:00:15.62 bitrate=6100.5kbits/s speed= 568x    
video:11379kB audio:240kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.154343%
bench: utime=0.003s stime=0.012s rtime=0.028s
bench: maxrss=26184kB

@Majestic7979
Copy link
Author

The physical hardware is the same, but there is a large difference in the amount of time to write a 10MB mp4 file. In my opinion this points to an issue outside of Frigate, and support here or Frigate itself does not know the details of the system on the host so it is very difficult to make specific suggestions on what to change.

I can give you any details you want if you ask me what you want to know

@Majestic7979
Copy link
Author

Here's something interesting

I cd into my ~ directory, which is a RAID1 array of 2x 16TB disks in mirror mode.
I did wget https://download.samplelib.com/mp4/sample-15s.mp4:
image

Then I cd into /cctv/test which is the new 4TB drive and ran the same wget (while Frigate container was stopped, of course):
image

If I do at my / which is a NVMe SSD this is the result:
image

Clearly, something is wrong because the /cctv volume is saving much slower, but this is a brand new disk. Could it be the way I mount it on fstab?

UUID=b963c953-82d9-4cf7-b6f6-xxxxxxxxxxxxxx /cctv ext4 discard,relatime,noexec,sync 0 2

How are your disks mounted? "discard, relatime, noexec, sync" is how mine is mounted.

@NickM-27
Copy link
Sponsor Collaborator

NickM-27 commented Aug 3, 2023

When I run findmnt this is what it shows

xfs rw,noatime,nouuid,attr2,inode64,logbufs=8,logbsize=32k,noquota

Other relevant info: Partition format: GPT: 4KiB-aligned

@Majestic7979
Copy link
Author

We come to an end with success - I would highly advise you to put in your documentation that the drive MUST be mounted with ASYNC parameter. The SYNC parameter was causing the issue.
The difference is dramatic and this has been a learning experience. My assumption was that forcing synchronous writes would ensure I have no data loss. It was with the intention to increase reliability. In the end, it just caused more issues. I would highly recommend you to ask users to check their fstab mount options going forward if there are performance problems and you think it might be related to storage. It might save you a ton of time.

2023-08-03 17:23:36.816862615  [INFO] Starting Frigate...
2023-08-03 17:23:37.789949504  [2023-08-03 17:23:37] frigate.app                    INFO    : Starting Frigate (0.12.1-367d724)
2023-08-03 17:23:37.799577066  [2023-08-03 17:23:37] peewee_migrate                 INFO    : Starting migrations
2023-08-03 17:23:37.958173792  [2023-08-03 17:23:37] peewee_migrate                 INFO    : There is nothing to migrate
2023-08-03 17:23:38.594006918  [2023-08-03 17:23:38] detector.tensorrt              INFO    : Starting detection process: 866
2023-08-03 17:23:38.606976682  [2023-08-03 17:23:38] frigate.app                    INFO    : Output process started: 868
2023-08-03 17:23:38.607060108  [2023-08-03 17:23:38] frigate.app                    INFO    : Camera processor started for frigate_street_front: 871
2023-08-03 17:23:38.607082494  [2023-08-03 17:23:38] frigate.app                    INFO    : Camera processor started for frigate_street_nw: 872
2023-08-03 17:23:38.607100227  [2023-08-03 17:23:38] frigate.app                    INFO    : Camera processor started for frigate_street_se: 873
2023-08-03 17:23:38.607117169  [2023-08-03 17:23:38] frigate.app                    INFO    : Camera processor started for frigate_backyard: 874
2023-08-03 17:23:38.607133566  [2023-08-03 17:23:38] frigate.app                    INFO    : Capture process started for frigate_street_front: 875
2023-08-03 17:23:38.607148450  [2023-08-03 17:23:38] frigate.app                    INFO    : Capture process started for frigate_street_nw: 878
2023-08-03 17:23:38.607480015  [2023-08-03 17:23:38] frigate.app                    INFO    : Capture process started for frigate_street_se: 883
2023-08-03 17:23:38.610699943  [2023-08-03 17:23:38] frigate.app                    INFO    : Capture process started for frigate_backyard: 889
2023-08-03 17:23:38.980533719  [2023-08-03 17:23:38] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init CUDA: CPU +189, GPU +0, now: CPU 245, GPU 246 (MiB)
2023-08-03 17:23:39.251186315  [2023-08-03 17:23:39] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 34 MiB
2023-08-03 17:23:39.251660009  [2023-08-03 17:23:39] frigate.detectors.plugins.tensorrt WARNING : Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
2023-08-03 17:23:39.616930818  [2023-08-03 17:23:39] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +193, GPU +74, now: CPU 501, GPU 356 (MiB)
2023-08-03 17:23:39.709895580  [2023-08-03 17:23:39] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +110, GPU +46, now: CPU 611, GPU 402 (MiB)
2023-08-03 17:23:39.711022590  [2023-08-03 17:23:39] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +34, now: CPU 0, GPU 34 (MiB)
2023-08-03 17:23:39.713953035  [2023-08-03 17:23:39] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 577, GPU 394 (MiB)
2023-08-03 17:23:39.713972199  [2023-08-03 17:23:39] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 577, GPU 402 (MiB)
2023-08-03 17:23:39.713991719  [2023-08-03 17:23:39] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 46 (MiB)
2023-08-03 17:23:54.104830337  [2023-08-03 17:23:54] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_backyard/23.40.mp4 in 0.1264801025390625 seconds.
2023-08-03 17:24:38.747959638  [2023-08-03 17:24:38] frigate.record                 DEBUG   : Start expire recordings (new).
2023-08-03 17:24:38.748125563  [2023-08-03 17:24:38] frigate.record                 DEBUG   : Start deleted cameras.
2023-08-03 17:24:39.482900574  [2023-08-03 17:24:39] frigate.record                 DEBUG   : Expiring 0 recordings
2023-08-03 17:24:56.343274553  [2023-08-03 17:24:56] frigate.record                 DEBUG   : End deleted cameras.
2023-08-03 17:24:56.343363128  [2023-08-03 17:24:56] frigate.record                 DEBUG   : Start all cameras.
2023-08-03 17:24:56.343416758  [2023-08-03 17:24:56] frigate.record                 DEBUG   : Start camera: frigate_street_front.
2023-08-03 17:24:56.587252838  [2023-08-03 17:24:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_street_front/23.40.mp4 in 0.12355995178222656 seconds.
2023-08-03 17:24:56.875086906  [2023-08-03 17:24:56] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_street_nw/23.40.mp4 in 0.12368893623352051 seconds.
2023-08-03 17:24:57.196630284  [2023-08-03 17:24:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_street_se/23.40.mp4 in 0.11715817451477051 seconds.
2023-08-03 17:24:57.279428240  [2023-08-03 17:24:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_backyard. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-03 17:24:57.280221740  [2023-08-03 17:24:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_front. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-03 17:24:57.280924589  [2023-08-03 17:24:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_nw. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-03 17:24:57.281624609  [2023-08-03 17:24:57] frigate.record                 WARNING : Unable to keep up with recording segments in cache for frigate_street_se. Keeping the 5 most recent segments out of 6 and discarding the rest...
2023-08-03 17:24:57.534087388  [2023-08-03 17:24:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_backyard/23.59.mp4 in 0.1273801326751709 seconds.
2023-08-03 17:24:57.812734040  [2023-08-03 17:24:57] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_backyard/24.09.mp4 in 0.12222504615783691 seconds.
2023-08-03 17:24:58.094206393  [2023-08-03 17:24:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_backyard/24.20.mp4 in 0.12455201148986816 seconds.
2023-08-03 17:24:58.412683381  [2023-08-03 17:24:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_backyard/24.30.mp4 in 0.1273789405822754 seconds.
2023-08-03 17:24:58.697981457  [2023-08-03 17:24:58] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_backyard/24.40.mp4 in 0.12827396392822266 seconds.
2023-08-03 17:24:59.015175372  [2023-08-03 17:24:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_street_front/23.59.mp4 in 0.12056708335876465 seconds.
2023-08-03 17:24:59.292716429  [2023-08-03 17:24:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_street_front/24.09.mp4 in 0.12518811225891113 seconds.
2023-08-03 17:24:59.594952060  [2023-08-03 17:24:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_street_front/24.19.mp4 in 0.11988306045532227 seconds.
2023-08-03 17:24:59.913355009  [2023-08-03 17:24:59] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_street_front/24.29.mp4 in 0.1272270679473877 seconds.
2023-08-03 17:25:00.175326061  [2023-08-03 17:25:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_street_front/24.40.mp4 in 0.12229108810424805 seconds.
2023-08-03 17:25:00.469362645  [2023-08-03 17:25:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_street_nw/24.00.mp4 in 0.12394905090332031 seconds.
2023-08-03 17:25:00.783117248  [2023-08-03 17:25:00] frigate.record                 DEBUG   : Copied /media/frigate/recordings/2023-08-03/16/frigate_street_nw/24.09.mp4 in 0.13649797439575195 seconds.

@Majestic7979
Copy link
Author

This is how it's mounted now, with sub-second copy times:

UUID=b963c953-82d9-4cf7-b6f6-xxxxxxxxxx /cctv           ext4    discard,relatime,noexec,async 0       2

@Majestic7979
Copy link
Author

I can confirm I now have a fully working installation with audio and no skips.

@Majestic7979
Copy link
Author

I feel confident about closing this issue now. I want to thank you for your time and patience and thank you and Blake for making this application and for supporting it. You are both top people.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants