Adds the ability for frigate to save clips of each tracked object

[blakeblackshear]

See readme updates.

[stoffus]

Looks amazing, and will most probably replace my recordings from within Home Assistant.

[jamescarlos]

Hi @blakeblackshear, this is really awesome - I tried out the save_clips container tag and for the most part it's working as expected. ffmpeg is writing to the /cache directory for all my enabled cameras and joins the files into an mp4 in the /clips directory when a person is detected.

Unfortunately, after some time (maybe several hours?) I run into a State file handle error when frigate tries to build a new clip:

frigate    | ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers
frigate    |   built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
frigate    |   configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
frigate    |   libavutil      55. 78.100 / 55. 78.100
frigate    |   libavcodec     57.107.100 / 57.107.100
frigate    |   libavformat    57. 83.100 / 57. 83.100
frigate    |   libavdevice    57. 10.100 / 57. 10.100
frigate    |   libavfilter     6.107.100 /  6.107.100
frigate    |   libavresample   3.  7.  0 /  3.  7.  0
frigate    |   libswscale      4.  8.100 /  4.  8.100
frigate    |   libswresample   2.  9.100 /  2.  9.100
frigate    |   libpostproc    54.  7.100 / 54.  7.100
frigate    | [mov,mp4,m4a,3gp,3g2,mj2 @ 0x55877f4ce400] Auto-inserting h264_mp4toannexb bitstream filter
frigate    | Input #0, concat, from 'pipe:':
frigate    |   Duration: N/A, start: 0.000000, bitrate: 1287 kb/s
frigate    |     Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuvj420p(pc, bt709), 1920x1080 [SAR 189:190 DAR 168:95], 1287 kb/s, 12.04 fps, 60 tbr, 12288 tbn, 24 tbc
frigate    |     Metadata:
frigate    |       handler_name    : VideoHandler
frigate    | /clips/front-1594778335.368959-0.mp4: Stale file handle

After this happens, I'll see errors similar to the above each time it tries to build a new clip from cache.
I do continue to get notifications through mqtt/homeassistant after this error, but no new clips are created until I restart the container.

Let me know if I can provide more information.

[blakeblackshear]

What did you mount at that path? Apparently, Stale file handle means that something happened to the mount at that location. Most of my searches turn up NFS related issues.

[jamescarlos]

Ah, that must be it. I bind mounted /clips to an NFS mount to my NAS on the host. I'll try it with just a host directory instead.

[mario-tux]

I'm trying to test this PR using docker container.
On startup I get a strange error:

...
eating ffmpeg process...,
ffmpeg -hide_banner -loglevel panic -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rtsp_transport tcp -stimeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp:https://XXX:[email protected]:80/videoSub -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an -map 0 /cache/camera-0-%Y%m%d%H%M%S.mp4 -f rawvideo -pix_fmt rgb24 pipe:,
Camera_process started for camera-3: 74,
Starting process for camera-3: 74,
Camera_process started for camera-4: 75,
Starting process for camera-4: 75,
Camera_process started for camera-2: 76,
Camera_process started for camera-1: 77,
Starting process for camera-2: 76,
Camera_process started for camera-0: 81,
Starting process for camera-0: 81Starting process for camera-1: 77,
,
 * Serving Flask app "detect_objects" (lazy loading),
 * Environment: production,
   WARNING: This is a development server. Do not use it in a production deployment.,
   Use a production WSGI server instead.,
 * Debug mode: off,
Exception in thread Thread-7:,
Traceback (most recent call last):,
  File "/opt/frigate/frigate/events.py", line 137, in run,
    event_type, camera, event_data = self.event_queue.get(timeout=10),
  File "/usr/lib/python3.7/multiprocessing/queues.py", line 105, in get,
    raise Empty,�
_queue.Empty,
,
During handling of the above exception, another exception occurred:,
,
Traceback (most recent call last):,
  File "/usr/lib/python3.7/threading.py", line 926, in _bootstrap_inner,
    self.run(),
  File "/opt/frigate/frigate/events.py", line 139, in run,
    self.refresh_cache(),
  File "/opt/frigate/frigate/events.py", line 42, in refresh_cache,
    start_time = datetime.datetime.strptime(f.split('-')[1].split('.')[0], '%Y%m%d%H%M%S'),
  File "/usr/lib/python3.7/_strptime.py", line 577, in _strptime_datetime,
    tt, fraction, gmtoff_fraction = _strptime(data_string, format),
  File "/usr/lib/python3.7/_strptime.py", line 359, in _strptime,
    (data_string, format)),
ValueError: time data '0' does not match format '%Y%m%d%H%M%S',
,

I can see the /cache folder filling up of small .mp4 files but nothing is created in /clips upon person triggering.

[blakeblackshear]

What are the file names in the /cache folder?

[mario-tux]

What are the file names in the /cache folder?

They look regular as expected:

camera-0-20200715221822.mp4  camera-0-20200715222049.mp4  camera-0-20200715222335.mp4  camera-0-20200715222607.mp4  camera-0-20200715222852.mp4  camera-0-20200715223123.mp4
camera-0-20200715221830.mp4  camera-0-20200715222101.mp4  camera-0-20200715222346.mp4  camera-0-20200715222616.mp4  camera-0-20200715222904.mp4  camera-0-20200715223132.mp4

Could it be related to the fact that my camera name contains a dash -?

[blakeblackshear]

That's the problem.

[mario-tux]

That's the problem.

Should I change my camera names or a more robust regex is going to be used in next commits?

[blakeblackshear]

I will fix it, but it won't be for a few days at least.

[jamescarlos]

Currently none of my clips contain an audio stream. Should we expect audio to be recorded?

[jamescarlos]

Currently none of my clips contain an audio stream. Should we expect audio to be recorded?

I think I got it by configuring additional input flags for audio streams.

[blakeblackshear]

Made a few improvements:

• You can define the number of seconds you want to capture before the event
• Use a random string in the filenames to support exposing public urls
• Push an updated MQTT snapshot when a better score is detected
• Fix issue with camera names that contain a '-'

Note the config format changed:

    save_clips:
      enabled: False
      #########
      # Number of seconds before the event to include in the clips
      #########
      pre_capture: 30

New image is blakeblackshear/frigate:save_clips-8e78760

[mario-tux]

Ok, thank you!
I understand that the last snapshot should be available on the MQTT channel. It would be useful to save (and update) the snapshot in the same place of the video, in the container mount, with the same filename and, obviously, different extension.

Is the filename of the video available among the MQTT message details?

[blakeblackshear]

You can build it from the data in the event message. It is {camera_name}-{id}.mp4. I'm still trying to decide the best way to save the image while minimizing the impact on performance. I use the camera.snapshot service to save within homeassistant's www folder as needed.

[mario-tux]

You can build it from the data in the event message.

Is there now a random token now?

[blakeblackshear]

Yes. It is in the id.

[rubenfiszel]

Would there be any easy way to modify this slightly to publish a clip on an mqtt topic as suggested by #169 ?

[rubenfiszel]

If we were to be able to publish the clip on an mqtt topic, would also be great to have an option to delete the file as soon as it is published to avoid filling the disk.

[mario-tux]

If we were to be able to publish the clip on an mqtt topic, would also be great to have an option to delete the file as soon as it is published to avoid filling the disk.

I think that this is not the right solution to this common problem. The mqtt protocol was not designed to transport such big payloads. In my opinion a simpler solution would be to include in the docker image of frigate a simple cron-like script that delete the files older than X-days from the clips folder. This cold be activated and configured using some environment variables to the container.

…

[rubenfiszel]

The mqtt protocol was not designed to transport such big payloads.

The maximum payload size seems to be 256Mb. Clips should be way below that so I do not really see an issue unless there are some documented limitations to avoid passing the video as mqtt payload.

Another solution is that instead of passing the file, one send a message with 2 urls as payload on that mqtt topic. One to fetch the video as a HTTP GET, the other one to delete the file with an HTTP DELETE. Requires to be able to serve files but given that frigate already uses flask it wouldn't be too hard to serve static files.

[jamescarlos]

Apologies if this is not the right place to ask.
Does this load look normal? I have 2 cameras currently saving clips using hardware acceleration. What are other users' CPU usage for ffmpeg and detect_objects.py using this image?

Specs:
Intel® Core™ i3-6100U Processor (3M Cache, 2.30 GHz)
8 GB RAM
500 GB SSD
Google Coral USB

[AT0myks]

The bounding box info is what I'm looking for, but is there a reason not to make it per object, as in frigate/<camera_name>/<object_name>/start? Sorry if I'm missing something, I'm new to MQTT.