Frigate, add masks to captured video?

[Fettkeewl]

Hello!

Thank you for an amazing NVR!
Frigate is awesome in all aspects. I'm so satisfied that I don't
have to rely on Blue Iris och Netcam and the likes.

I have a question:
Here in Sweden, we got strict laws in regards to private camera surveillance.
In short we are allowed to surveil our own property and nothing else, that means a camera that
shows even a part of the public sidewalk is to be considered illegally installed.

Now I have a camera which I can't angle any better than it is already to minimize the public areas seen.
I was wondering if it was possible to apply the masks we create for "non-detect" areas to the video that is recorded
when motion is detected. So basically making some areas black as a part of the video...

With regards!

This area would have to be blackened in captured video

[blakeblackshear]

In order to modify the recorded video, you will need to re-encode. You can use ffmpeg to overlay images on the video. This will take a lot of CPU resources, so I would suggest replacing the camera with one that can add these privacy filters to the stream directly.

[Fettkeewl]

In order to modify the recorded video, you will need to re-encode. You can use ffmpeg to overlay images on the video. This will take a lot of CPU resources, so I would suggest replacing the camera with one that can add these privacy filters to the stream directly.

Ah .. was worth a shot, I figured that might be an issue. Thank you for your time :)

[antan951]

Hello
I also need to mask some areas of the camera feed to comply with local rules and the cameras I already acquired are not up to the task. was hoping that it would be possible to mask at least the reduced resolution feed that is used for detection and did some trials outside of frigate.

This command sequence seem to do the trick:
ffmpeg -loglevel info -hide_banner -c:v h264_qsv -i 'rtsp:https://username:password@camera_ip:554/cam/realmonitor?channel=1&subtype=2' -i triangle-overlay.png -filter_complex 'overlay' test.mp4
I tried both on a local computer and from within the docker container and they both produce a video file that has been correctly masked.

Then I tried to "fool" Frigate into doing the same by squeezing the mask into the - path: section for the camera.
path: rtsp:https://username:password@camera_ip:554/cam/realmonitor?channel=1&subtype=2 -i /media/frigate/triangle-overlay.png -filter_complex 'overlay'

Unfortunately it does not seem to work. Frigate starts without anything unusual in the log, but the mask has not been applied.
In the debug section, I see this ffmpeg cmd: "cmd": "ffmpeg -loglevel info -hide_banner -c:v h264_qsv -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp:https://username:password@camera_ip:554/cam/realmonitor?channel=1&subtype=2 -i /media/frigate/triangle-overlay.png -filter_complex 'overlay' -r 5 -s 1280x720 -f rawvideo -pix_fmt yuv420p pipe:",
This looks quite good to me, and it seems that the additional arguments survived the configuration parser.

Any advice how to proceed would be highly appreciated.

[ProfessorBoemmel]

Hi everyone ..

so i got it working (following examples in [https://hhsprings.bitbucket.io/docs/programming/examples/ffmpeg/] )
in an external and also the internal go2rtc - hope it helps.

If you see improvement, please let me know.

Now i think i need to live with the CPU used
and to play with the pre_ and post_capture parameters to fight the delay i might have due to the pre-processing.

go2rtc:
  streams:
    mycam: 
      - rtsp:https://YOURACCOUNT:YOURPASSWORD@YOURIP/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
    mycam_small: 
      - rtsp:https://YOURACCOUNT:YOURPASSWORD@YOURIP/cam/realmonitor?channel=1&subtype=1&unicast=true&proto=Onvif
    mycam_small_blur: 
       - ffmpeg:mycam_small#video=h264#raw=-i /media/frigate/masks/mycam_small_mask.png -filter_complex "[1:v]loop=-1:size=2,scale=320:240,setsar=1[alpha];[0:v]boxblur=10[blur];[0:v][alpha]alphamerge[vt];[blur][vt]overlay"
    mycam_blur: 
       - ffmpeg:mycam#video=h264#raw=-i /media/frigate/masks/mycam_big_mask.png -filter_complex "[1:v]loop=-1:size=2,scale=1280:720,setsar=1[alpha];[0:v]boxblur=10[blur];[0:v][alpha]alphamerge[vt];[blur][vt]overlay"

cameras:
  mycam:
    enabled: true
    ffmpeg:
      inputs:
      - path: 
          rtsp:https://127.0.0.1:8554/mycam_small_blur
        roles:
        - detect
      - path: 
          rtsp:https://127.0.0.1:8554/mycam_blur
        roles:
        - record

[deanpm70]

Fantastic news! I'm going to try this myself this evening. Will report back with progress.

[deanpm70]

So that works. I'm using a straight masking overlay rather than a blur but the key was the format which you provided. My go2rtc config is as follows;

    CAM02_MASKED:
      - ffmpeg:CAM02#video=h264#raw=-i /media/frigate/CAM2-MASK.png -filter_complex "[0][1]overlay=x=0:y=0" -c:v hevc_nvenc

I think the -c:v hevc_nvenc at the end is ignored judging by the cpu high cpu usage but that's a problem for tomorrow.

[ProfessorBoemmel]

Thanks @deanpm70 Definitely a simpler filter that should also release some CPU usage.
Somehow I was hoping that the internal go2rtc or ffmpeg would also be able to utilize the edge CPU (coral usb) i have attached, but sadly not the case.

[deanpm70]

I’m very close (I think) to getting this running on my Nvidia gpu using the overlay_cuda filter. I’ve got it running on the command line inside the frigate docker (on an Ubuntu 22.04 host) but there’s something about the output it generates that frigate doesn’t like.

[hotzen]

wow, this really is a bummer I have to say. the privacy laws in germany (and generally EU) are very strict about this.
is ffmpeg encoding really the only way forward here?

[NickM-27]

Yes, it's required to re-encode if your camera doesn't support it, there is no way around that.

[hotzen]

thanks for the clear response

[ProfessorBoemmel]

Hi everyone,

sadly i have the same situation with a "dumb" (but way to expensive :D) doorbell - that films visitors and sadly, the street and houses behind them.

That said even "static" bluring will not help and i am wondering if its even possible to get the data of the bounding boxes of the detected objects while the record is taken + an exit trigger so a post script could blur out everything except the bounding boxes ?

Would it be worth to raise this as feature request, or is ist just impossible.

I managed to blur "the top half" (street and houses) but that also means the detected person will never be recognizable when standing infront of the doorcam.

Thanks i.A.

[deanpm70]

I'm in the same exactly the same situation. Does anyone have an actual working example of a config that uses ffmpeg to apply an overlay image?

[ProfessorBoemmel]

Sadly no.

I managed to get some local tests with ffmpeg working.

But as i fail to get filter_complex accepted in frigate output args ...

ffmpeg: output_args: -filter_complex ...

... always get the complain that -vf not compatible with -filter_complex (due to -vf r=5,scale=.. already present by hwaccel i think)
i am currently experimenting with the internal and an external go2rtc and a simplified config like:

streams:
cam:
- rtsp:https://user:password@ip/cam/realmonitor?channel=1&subtype=0&unicast=true&proto=Onvif
cam_small:
- rtsp:https://user:password@ip/cam/realmonitor?channel=1&subtype=1&unicast=true&proto=Onvif
cam_small_blur:
- ffmpeg:cam_small#video=h264#raw=-filter_complex "[0:v]crop=320:160:0:0,boxblur=15[blur];[0:v][blur]overlay"
cam_blur:
- ffmpeg:cam#video=h264#raw=-filter_complex "[0:v]crop=1280:480:0:0,boxblur=15[blur];[0:v][blur]overlay"

and then use those inputs accordingly:

mycam:
enabled: true
ffmpeg:
inputs:
- path:
rtsp:https://go2rtc:8554/cam_small
roles:
- detect
- path:
rtsp:https://go2rtc:8554/cam_blur
roles:
- record

Seems i need to play with the pre and post captures as between object detection and the "blurred video" there is some delay.

Hope to find a better solution with frigate only,
and for me best would be an option "blur on record" all that is not in the object boxes.