[Support]: Configuration of HWaccel, multiple TPUs and general performance questions #12399
-
Describe the problem you are havingI've been using/testing Frigate for a while now to get acquinted, and I like the project and the direction it seems to be headed. I think parts of my setup are less-than-ideal though, and would like some help in improving my config. Goals
It's also worth noting, that while I currently run 5 cameras total (3x8MP, 2x2MP) the end goal is going to be around 30 cameras with the majority being 2MP and maybe a total of 5-7 8MP cameras. As per #2 above, I'll be wanting a good number of these cameras (probably 25+) to be involved with various automations, like triggering lights, automatic doorbell, etc. Hardware/Software
As for the physical specs (the above are somewhat meaningless on their own):
I've tried to create the setup so that I shouldn't run into any ressource constraints with Frigate, with the intention of subsequently optimizing usage based on reported stats and your input. However, Frigate doesn't really seem to run all that well yet, so I haven't gotten to that part yet. I believe that's all of the important stuff out of the way - let me know if I missed something - on to questions. Question 1 - Hardware Acceleration Should I be looking into installing a dedicated graphic card in this server, also with respects to the number of cameras I intend to install? Or "should this work" given a proper setup for CPU based HWaccel? If so, can anyone point me in the right direction? Caused by no physical GPU being present in the system (neither on chip or dedicated). Two GPUs installed (the best/most this server can physically accept without a difficult to source enablement kit). The GPUs are passed through Proxmox to the VM (search for this topic separately on Google if you need to) and the Docker Compose is changed as per the documentation. See this post for manual load balancing and configuration of Frigate. Question 2 - TPUs The way I've added these is pieced together mainly through ProxMox forum posts, but to paraphrase, I've gone to Proxmox/VM/Hardware and added two USB devices which refer to the physical port (4-5 and 4-6 in this case). I would have liked to refer to vendor/device ID but my Corals report the same ID and the Frigate config doesn't accept identical entries, IIRC. Within Frigate, I then refer to them through usb:0 and usb:1, which as I mentioned worked, but not persistantly. So I'm running on 1 Coral TPU and have two idle. I'd love to get all three online, but don't see how to progress? They are obviously plugged into USB3 ports, but if it's a pointless endeveour and I'm much better of using a PCI Coral, then perhaps that's what I should be doing? That requires a PCI riser to be installed in the server, and for that reason as well as for them being more universal, I originally went with the USBs. I also seem to recall having researched user opinions and comparisions that found that there were (then) almost no disernable difference in inference speed with PCI vs USB options. This has solved itself, likely though a Proxmox update. I reenabled the second TPU in the frigate config and it is reliably found according to the logs. The systems page displays 2x TPUs. I will possibly add the third TPU in the future when more cameras are installed. Question 3 - General performance and stability _Logs to be reviewed if it occurs again. The issue hasn't occured again, and I assume it is caused by a defective camera disturbing Frigate too much. See this post for details. _ Question 4 - Recommendations for cameras Should I look into the options in the documentation, or are there perhaps more up-to-date recommendations? Stability and image quality is more important than price, availability in Europe is essential and I don't want to pay more for the camera than I need to (i.e. to buy for example a Hikvision branded camera at 2x the price of an identical Dahua branded camera that it's based on or whatever). Processed. Question 5 - Secrets Solved by: #12399 (reply in thread) I realise this is a super-long list of questions, but figured it would make sense to combine all the questions I have into one thread to ensure all relevant logs and specs are present as well. If this is totally outrageous, please let me know. :) Thanks. KR, Version0.13.2 - 6476F8A Frigate config filePlease note that this is not the final Frigate config file but rather the original. The thread contains changes made to it. mqtt:
host: mqtt.bmsip
topic_prefix: frigate
detectors:
coral1:
type: edgetpu
device: usb #:0
# coral2:
# type: edgetpu
# device: usb:1
# ov:
# type: openvino
# device: CPU
# model:
# path: /openvino-model/ssdlite_mobilenet_v2.xml
cameras:
CAMe03-West:
ffmpeg:
inputs:
- path: rtsp:https://user:[email protected]/live
roles:
- record
- path: rtsp:https://user:[email protected]/cam/realmonitor?channel=1&subtype=2
roles:
- detect
objects:
track:
- person
- car
- bicycle
- motorcycle
motion:
mask:
- 1308,98,1861,96,1864,40,1311,42 # Timestamp upper right, 1920x1080.
zones:
ext_person_west_near:
coordinates:
0,380,290,1080,1920,1047,1920,503,1638,375,1296,219,1018,121,646,24,433,90,160,152,0,124 # Area near building
objects:
- person
ext_person_west_far:
coordinates: 779,67,775,0,1206,0,1293,108,1920,117,1920,513,1528,309,1213,201,955,111 # Driveway, across field road
objects:
- person
ext_vehicle_west_driveway:
coordinates: 772,173,1920,834,1920,479,1339,198,1193,55,902,0,569,106 # Driveway and field road
objects:
- car
- motorcycle
- bicycle
ext_vehicle_west_near_entry:
coordinates: 670,183,0,364,0,232,537,120 # From field road to entry
objects:
- car
- motorcycle
- bicycle
detect:
width: 1920
height: 1080
CAMe04-NorthWest:
ffmpeg:
inputs:
- path: rtsp:https://user:[email protected]/live
roles:
- record
- path: rtsp:https://user:[email protected]/cam/realmonitor?channel=1&subtype=2
roles:
- detect
objects:
track:
- person
- car
- bicycle
- motorcycle
motion:
mask:
- 1308,98,1861,96,1864,40,1311,42 # Timestamp upper right, 1920x1080.
- 1920,0,1920,178,1920,284,1857,357,1812,347,1766,299,1703,234,1652,169,1408,123,1373,127,1273,150,1171,160,1127,200,940,262,879,298,829,297,746,229,659,50,615,0 # Vegetation, Sky and Neighboor
zones:
ext_person_north_near:
coordinates:
321,1080,1920,1080,1920,1047,1769,972,1449,829,1212,728,967,638,716,567,474,509,241,455,74,415 # Area near building
objects:
- person
ext_person_north_far:
coordinates:
729,290,994,382,1290,105,1814,322,1874,314,1920,314,1920,878,1617,810,1371,732,1118,653,851,578,598,512,321,452,560,364 # Storage, Parking, Pens and Area further away than middle of road.
objects:
- person
ext_vehicle_north:
coordinates: 328,1080,1920,1080,1920,394,1206,209,101,505 # Storage, Parking and General Area
objects:
- car
- motorcycle
- bicycle
detect:
width: 1920
height: 1080
CAMe05-NorthEast:
ffmpeg:
inputs:
- path: rtsp:https://user:[email protected]/live
roles:
- record
- path: rtsp:https://user:[email protected]/cam/realmonitor?channel=1&subtype=2
roles:
- detect
motion:
mask:
- 1308,98,1861,96,1864,40,1311,42 # Timestamp upper right, 1920x1080.
- 244,310,322,345,367,418,470,385,479,341,595,310,813,260,834,212,860,149,903,151,1608,173,1623,204,1662,224,1688,330,1707,380,1750,389,1768,364,1852,267,1920,203,1920,0,0,0,0,316 # Vegetation, sky, field
zones:
ext_person_north_near:
coordinates:
1695,1080,0,1080,0,1042,0,952,30,950,361,953,736,939,1034,897,1190,855,1329,829,1436,819,1536,818,1596,715,1688,565,1742,506,1899,525 # Ramp, area near building
objects:
- person
ext_person_north_far:
coordinates:
959,874,1142,739,1261,790,1389,822,1532,819,1577,741,1643,634,1733,522,1700,419,1705,369,1920,406,1920,346,1057,285,0,617,0,904 # Dump, area near pens, road, area further from building.
objects:
- person
detect:
width: 1920
height: 1080
CAMi01-Workshop:
ffmpeg:
inputs:
- path: rtsp:https://user:[email protected]/live
roles:
- record
- path: rtsp:https://user:[email protected]/cam/realmonitor?channel=1&subtype=2
roles:
- detect
zones:
int_person_0W03_workshop_all:
coordinates: 0,720,1280,720,1280,0,0,0 # Full camera frame for detection area 1280x720.
objects:
- person
int_person_0W03_workshop_desk:
coordinates: 705,720,780,639,559,469,554,219,230,95,0,219,0,720 # Area near desk and conference section.
objects:
- person
detect:
width: 1280
height: 720
CAMi07-Storage:
ffmpeg:
inputs:
- path: rtsp:https://user:[email protected]/live
roles:
- record
- path: rtsp:https://user:[email protected]/cam/realmonitor?channel=1&subtype=2
roles:
- detect
motion:
mask:
- 961,61,1237,61,1234,21,959,22 # Timestamp upper right on detect resolution 1280x720.
zones:
int_person_0W02_storage_all:
coordinates: 0,720,1280,720,1280,0,0,0 # Full camera frame for detection area 1280x720.
objects:
- person
int_person_0W03_workshop_all: #turn on lights in workshop before entering
coordinates: 1151,286,1082,201,1001,224,970,288,1056,432,1144,386 #Area in front of door to workshop
objects:
- person
detect:
width: 1280
height: 720
birdseye:
width: 1920
height: 1080
quality: 10
mode: objects
record:
enabled: true
retain:
days: 7
mode: all
snapshots:
enabled: true
timestamp: true
bounding_box: true
retain:
default: 30
live:
height: 1080
quality: 10
# ui:
# use_experimental: trueRelevant log output(None for now)FFprobe output from your camera(None for now)Frigate statsNo response Operating systemDebian 11 Install methodDocker Compose Object DetectorCoral Network connectionWired Camera make and modelDahua, various models Any other information that may be helpfulNo response |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 30 replies
-
|
Beta Was this translation helpful? Give feedback.
-
|
RE: 2 ... I've not had any issues with using 2 USB Corals passed into a Proxmox LXC (although I'm currently using just one because it's sufficient for my use case). With that said, I do believe the problems you are experiencing are solvable. |
Beta Was this translation helpful? Give feedback.
-
I think this was the issue, in combination with a few others (like the cameras seemingly having some brute-force protection that kicks in and takes quite a while to cool down, thus camouflaging the working solution by continuing to refuse connection). But it's working as expected now, thanks. :) In regards to # 1 I obviously need to install a proper GPU, however with this being an enterprise server, there are likely a tonne of restrictions not readily apparent to me. I imagine the following constraints are particularly relevant:
I'm hoping the IT supplier and general research can help with all that. But what neither the IT supplier or me are likely knowledgable of, is what exactly I should be looking for? Can you @NickM-27 clarify, keeping the goal number of cameras, etc. in mind, what I should be looking for? Thank you so much. Edit: While I can't find any official documentation (any longer), I did find a Reddit post that refers to the options available if that helps, but still unsure which priorites I should have for Frigate. A link to the post is here for reference: https://www.reddit.com/r/sysadmin/comments/xemm3n/comment/ioipn76/ |
Beta Was this translation helpful? Give feedback.
-
Thanks :) This is a VM, not a LXC because my Proxmox instance is using ZFS. I recall that there are differences, although the setup is likely similar. Can you clarify if your 2 USB corals appeared with identical or different DeviceIDs? I believe I would have no issues if not for the fact that the deviceIDs on two of my Corals are identical, and as such I am forced to instead reference them by port number, which is apparently not guaranteed to be persistant across reboots. |
Beta Was this translation helpful? Give feedback.
-
|
for the usage discussed here you just need something that supports decoding and maybe encoding if you wanted to transcode some stream for some reason. You would be looking for pretty much any nvidia or amd discrete card that supports decoding of the types of streams you want to work with. Intel is technically also an option but linux kernel support is much more limited to newer versions |
Beta Was this translation helpful? Give feedback.
-
It happened again today. As mentioned, one of my Dahua cams (CAMe03) simply does not work, and in hindsight I should probably comment it out of the config until it's replaced. For one, it clutters the log a lot, and I guess this issue could somehow be related, i.e. that Frigate can't keep up with it's other jobs due to being tied up in some time-out event on this camera? I'll comment it out regardless, but I'll still post the logs up below in case it's a completely unrelated issue. Cameras page: (CAMe03 is "expected" to show no frames. The other four cameras are unexpected as they normally work. System page: Excerpts from log (due to clutter from CAMe03). Frigate starting: Immediately after starting, these errors "loop" on a 10s interval (again; this is expected, I'm including it in case something important is in it): After around 20 minutes (but normally the cameras can work for days and weeks without the error also), this line is "appended" to the above loop, so it's also repeated every 10 s. Initially, it only complains about CAMi01: A further 11 minutes later, CAMe05 starts showing up, but this is more intermittent. I shows up in 2-3 loops, but then drops back out leaving only CAMi01 in the repeating loop: Then, another 11 minutes or so later, CAMi07 is included along with CAMi01, but no other cameras: And like before, CAMi07 will occasionally drop from the log, only to return later, etc. Eventually the log (including the irrelevant snippets from CAMe03) loops like this: And while no camera has reliably been showing the recording segments warning reliably (even CAMi01 disappears from the log over large spans of time), no cameras actually work when viewing the camera page. The latest log excerpt is from 12:07 today, and only shows issues with CAMi01 and CAMe05, but as shown in the screenshots, other cameras are also showing no frames. As mentioned, I'll comment out CAMe03 untill it's replaced, but other than that I don't really know what to do. I also don't know if this error is likely caused by CAMe03 or if it's a separate issue that should be looked into? |
Beta Was this translation helpful? Give feedback.
-
|
After the camera is commented we really need a continuous segment of logs not pieces to see what happens |
Beta Was this translation helpful? Give feedback.
-
I already commented the camera out and rebooted (which also "fixes" the error), so I'll see if it comes back, and if so post a continuous log. However, just to be clear, there's nothing going on in the log apart from the above looping segments. :) I did look through it quite carefully. How did you change the last log segment to "look properly"? Quote instead of code? And thanks for the tips on GPUs, I'll ask my vendor. |
Beta Was this translation helpful? Give feedback.
-
|
Use 3 back ticks instead of 1 at the beginning and end |
Beta Was this translation helpful? Give feedback.
-
|
Hi again @NickM-27, I bought two cards (they were not expensive) and have them both successfully passed-through into the VM that hosts Frigate as a PCIe hardware device pass-through. However, I am very confused by which driver to install. Currently, both cards use the Nouveau driver. From the link in the docs to ffmpeg driver version requirements, it seems I must use the Nvidia driver, and it seems it must be version 550.54.14 or newer. Relevant output from my system: However, I'm on Debian 11. And according to the wiki (link below), only up to 470 is supported on Debian 11 and only up to 535 on Debian 12 or Debian Sid (unstable). I get that I can apply backports (haven't ever tried that though), but surely it can't be right that ffmpeg requires bleeding edge drivers that can't be installed on stable OS's for something so simple? I must be misunderstanding something? Can you help advice what I'm supposed to be installing driver-wise? :) Link to the wiki page mentioned: https://wiki.debian.org/NvidiaGraphicsDrivers |
Beta Was this translation helpful? Give feedback.
-
Oh, right. Gotcha, thank you. I've read the entire thread, and step-by-step am applying it to my configuration, since go2rtc makes a lot of sense in general. You can see my config below. Note that I have not yet saved this last bit, so the lines gpu0 and gpu1 are not yet applied. To my very untrained eye, the example linked above looks to revolve around taking 1 main stream from the camera and copying+resizing it to make it suitable for the detect role? I already pull two streams from each camera, so I don't need that bit, but in my understanding, still can benefit from hwaccel for Frigates decoding and processing of the substream. At least I assume that is the point of all this. :) So I'm a bit puzzled how to correctly apply hwaccel in my scenario? What would I typically put in the back_porch_sub of the example / my substream for detect to utilize hwaccel correctly? And is there any way to see the "expanded" list of arguments that the preset applies? I might as well build it into the arguments under streams/ffmpeg, but looking in System/Camera/ffmpeg I'm given a bunch of stuff also, which I assume Frigate adds/implies, rather than what I'd actually need to enter. The relevant parts of my current config: |
Beta Was this translation helpful? Give feedback.
-
|
No, you can't use go2rtc to change what hardware is used inside of Frigate itself. This would be done by setting |
Beta Was this translation helpful? Give feedback.
-
Sorry for being slow - I'm unsure where exactly the above would go. Am I right to suggest that I then remove ffmpeg: near the top of my config (or does that stay?) and then what you write above goes in my config as exemplified below, and the section with go2rtc streams stay, but contain no ffmpeg stuff, only rtsp paths in my case: Thank you so much for your patience. :) |
Beta Was this translation helpful? Give feedback.
-
|
the only thing that changes about your config is setting the global args for each camera. Half using 0 and half using 1. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Nick, I'll update the main post to the benefit of anyone searching in the future, and select one of your posts as answer. For anyone as dim as myself, this is the configuration that worked for me, pertaining to go2rtc, restreaming via localhost and manually load balancing hardware accel: To verify settings, you can run nvidia-smi on the host (outside the docker container): |
Beta Was this translation helpful? Give feedback.
for the usage discussed here you just need something that supports decoding and maybe encoding if you wanted to transcode some stream for some reason. You would be looking for pretty much any nvidia or amd discrete card that supports decoding of the types of streams you want to work with.
Intel is technically also an option but linux kernel support is much more limited to newer versions