Frigate For Commercial Scale

[JShimane]

Hello! I am looking to use Frigate for around 60 cameras, with detection. I'm hoping to figure out what kind of hardware that would require. It's preferred that I use a single central server.

Is there any sort of rule/suggestions for hardware per camera?
i.e., TPU/cameras, CPU/cameras, GPU/cameras

Or is there any way to figure that out based on FPS and resolution per camera?
Thank you!

[NickM-27]

There are 3 main sources of resource usage in Frigate, in no particular order: object detection, decoding video, motion detection

object detection

Frigate supports a number of different object detection accelerators from Google Coral to Intel & Nvidia iGPU / GPUs

Frigate runs motion detection and once motion is detected, Frigate will run object detection on the motion area(s) (referred to as regions). The object detector has a model which expects a specific size image (for example the default coral model expects 320x320 images). For this reason, the resolution of the camera does not affect the inference speed. Inference speed is the main concern with object detection performance because Frigate runs in real time and if inference speed is too slow, will skip running object detection on some regions in order to keep up with real time.

So, if you have an inference speed of 10 milliseconds then that means you can run (1000 ms in a second / 10 milliseconds per inference) 100 object detection inferences per second. Frigate often runs multiple inferences on a single camera frame, for instance when motion happens in two places at the same time, when an object was partially detected at the edge of a region and a larger region should be used, etc.

The default fps for object detection is 5, this is generally good for most cases but there may be some cameras that need closer to 10 if there are frequent fast moving objects.

decoding video

Frigate must decode the cameras video stream in order to capture each frame for motion + object detection. The docs highly recommend running detection on a sub stream of the camera given that regions are resized and there is in most cases no benefit to running object detection on higher resolutions. In my personal opinion 720 x 1280 is a good starting point, some cameras may need to use a bit higher if you are trying to detect small objects on that camera and it is not working consistently. Using ffmpeg, frigate supports recent AMD, Intel iGPU, and AMD iGPU / GPU. Those GPUs will have different specs that show how many concurrent stream decodes it supports as well as at what resolution.

motion detection

Frigate runs motion detection to intelligently know when and where to run object detection. Motion detection is only run on the CPU and is directly affected by the resolution of the detect stream, which is another reason that it is recommended to run a sub stream for detect.

[Cold-Lemonade]

@JShimane As @NickM-27 explains and your last question indicates, the answer depends not only on the number of cameras you intend to use, but also their resolution, fps, and how long you want to save the recordings.

I have 8 cameras. For detection, I use their substreams at 704x480 at 5 fps. For recording, I use their main streams which are a combination of 2688x1520, 2048x1536, and 1920x1080 at 20 fps, 25 fps, and 30 fps. I am running Frigate on a mini pc with an Intel n100 cpu and 16 gb of ram, 2 coral USBs, and 10 TB hard drive. This pc also runs Home Assistant in a vm. When I check htop to see my system utilization, the cpu is usually not more than 50% utilized and over half the ram is not used either. I save recordings for 14 days and use less than half of my hard drive.

[marleyjaffe]

are both coral USB's required? Were you experiencing issues before you added the 2nd?

[Cold-Lemonade]

@marleyjaffe I am not sure whether the 2nd coral USB makes any difference for me. I have noticed no change in performance with the second one. But I plan to add a few more cameras.

[marleyjaffe]

Makes sense. Thanks for the info

[YBonline]

I have 30 cameras of varying qualities/ages, encodings (h265 and h264), and styles (panaromic).

I ended up having to upgrade a bit of hardware to keep up. I ended up buying a Dual Xeon 4208 server (Dell R440 refurb) and Frigate seems to use about 40GB of my 62GB of RAM. I had to add a video card since the Xeons didn't support hardware decoding of the video, and I had to return my first one as it didn't have enough RAM, so thats probably the biggest thing to think about: your gonna wanna store all the clips in RAM while they're being processed, and your gonna want a video card to decode the videos, so your going to want a TON of RAM and video RAM, that seems to be the most important spec. I ended up with a NVIDIA T1000 8GB. I'm not using much GPU, that seems to stay around 10%, but my server only could take a single slot, relatively small card, and that seemed to be my best value (I also didn't know exactly how much GPU power I would need, turns out not much)

I did run my 30 cameras successfully on one Google Coral USB, but I had a dual edge TPU (the mini PCI Express one) on order for a while and it came in a few weeks ago (along with an adapter that supports dual edge TPUs). While the USB had no problem keeping up with my cameras, when I switched over to the dual edge TPU, I had DRASTIC drops in memory usage, as Frigate was able to pull the videos out of cache that much faster. My inference rate went from 20ms to about 8ms on average, which seems insignificant, but that seemed to lead a several GB drop in memory usage. Infact, I felt my 64GB of RAM wasn't enough and thought I couldn't put more cameras in prior to getting the dual edge TPU., but now I feel I can definitely squeeze some more in.

I've been reporting my issues with my larger then normal number of cameras to Nick, and he has been absolutely amazing at getting them fixed for me! But I will note, the fixes are all in the dev branch.... so your going to need to run that until 0.13 comes out. There are a bunch of multithreading improvements, DB performance improvements, etc in the latest dev builds that are, IMO, critical to an implementation you describe.

For storage, I'm currently keeping 4TB of video. I keep my database on a separate hard drive from the recordings. In fact, I'd recommend keeping the recordings on their own drive. I seem to be currently producing about 35GB/hour of recordings, but I've been meaning to try to adjust motion sensitivity on some cameras as they end up recording 24/7. Well, and my 2 LPR cameras that I just got recently are also set to record 24/7

If I was doing this again, I'd probably choose a different server that can handle multiple video cards.

If you have any other questions, let me know

[eyajuda]

@YBonline,
Incredible, setup and experimentation you had. kudos, I am new to Frigate, been trying my hand now with cameras progressively to from 1, 3, 4, approaching 10. I have have a core i5, with iGPU, with 8GB ram, Coral USB TPU, running on ubuntu 22 bare metal, my goal is to wire a large compound with about 60 cameras, want to be able to do some analytics, facial image etc, had read a lot about the various possibilities on frigate. Can you help me with you configuration setup file to help my learning and experimentation process also to help reduce my trial by error period?
will sincerely appreciate this, and will appreciate any advice you can offer.

[YBonline]

Sure! I actually haven't put too much more effort in since that last post, I think most of that still applies. 0.13 Beta 3 is out now, make sure to start with that, don't try the 0.12 stable branch as you're going to need the optimizations in it. Nick also is super helpful, probably way more helpful then me!

I haven't increased my cameras since that last post, and am going to be too busy for the next couple months, but I plan to move my final standalone NVR to Frigate early next year (which I think will bring me to 36 cameras total, and I have no near term plans to expand beyond that, at least until I get fiber run to more locations).

The other thing I've learned through this is when you're setting up a large number of cameras, you want to have different purposes for different cameras. You'll want some overview cameras that cover large areas, but don't try to really see who is who with them. You'll want to have some very low cameras aimed at the entrances to capture faces for facial recognition. You'll only really want your cameras specifically designated to capture faces at high resolution, everything else should be at low resolution for detection/object recognition to avoid unnecessary resource use. Ceiling mounted cameras don't work well for that. Likewise, if you want to capture license plates, you'll want cameras setup specifically for that and nothing else. I haven't messed with fisheye cameras or PTZ cameras, but they really all have different purposes, so make sure you understand that when laying out your cameras.

[jeremyjoslin]

If I were to buy a GPU on eBay what matters most? 16GB RAM? There’s one for $75 right now.

[YBonline]

yeah, gpu ram seems to be the only thing that matters from my experience, other resource consumption has been extremely low on my nvidia t1000

[GrahamCrackrr]

I have an installation running 12.1 using an Nvidia 3090 with 24GB of video RAM for detection.

There are 36 Reolink cameras in an office space. Detections are on the main stream using:
width: 1280
height: 720

The single Nvidia card appears to be sufficient to the task and the system is running well.

.

From what I could tell, 12.1 wont allow you to run detection on multiple video cards currently. 24GB of RAM is the biggest Nvidia card right now, so if we wanted to run detection on a larger qty of cameras, we would need to run detection on a lower resolution I assume. If V13 supports multiple video cards, then it would open up a lot more potential for a larger qty of cameras.

[thunderwolf66102]

Would ZFS offer any performance increases given that I have 96 GB of RAM to play with? 4 spinning platters of rust of course, no SSDs here. And would I have to break the array to get the benefits?

Though I am wondering if it is just the limit of how fast ffmpeg can copy the segments from cache to storage. That, and there still may be errors that I have not been able to catch yet that are causing processing to fall behind?

[NickM-27]

the best bet is to just enable debug logs for frigate.record.maintainer and see what the times actually are as the errors start to occur

[GrahamCrackrr]

I also only record on motion as well on Frigate. The only issue I have seems to be how long it takes to filter down those recordings. That appears to work VERY slowly. I have a separate 8 bay Synology unit that records everything everything.

[YBonline]

Just curious, you using H265 for the bandwidth savings? Thats something that also massively helps out. Also how much data are you actually producing? 800MB/sec sounds like it should handle 18 cameras without issue based off my cameras data.

[thunderwolf66102]

For those Reolink cameras that only offer h265 for the full resolution streams I convert it to h264 via restreamer and pull it to go2rtc/frigate from there. It would probably help if I just left these feeds h265 but I want to be able to access these feeds on Windows, Linux, Android and iOS.

I did tune the motion masks on some of the cameras, particularly those that have a major road in view. These are also some of the high resolution cameras, including a Reolink Duo 2 POE. I've avoided the 'Unable to keep up with segments' today, but still caught the cache ballooning up to over 1 GB, sometimes moments later dropping back down to just over 100 MB.

I think 400-500MB is probably more realistic sustained as 500MB was the result before tweaking various sysctl values. So am I exceeding or bypassing the kernel's cache with frigate? How does ffmpeg write the files to disk? Are the writes direct, buffered, SYNChronous or not? Looking into ZFS it seems like I could better leverage the RAM that I have by setting sync=disabled making all writes asynchronous?

I'm just running docker under Ubuntu on bare metal. No supervisors.

How hard would it be to have Frigate dump certain groups of cameras to a different location other than the default/usual?

[YBonline]

Also make sure your cache is really on tmpfs, until a recording is saved you really need it stored in RAM. Portainer and other such "docker supervisors" can cause the tmpfs not to be stored in RAM. I experienced that at a previous point before i realized what was going on.

ffmpeg should never be writing to anything but RAM from my understanding.