-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Support]: I keep getting "unable to keep up with recording segments" and my footage skips a lot. #7297
Comments
This is the camera details - my camera is basically the same one, but mine has 4x optical zoom. They don't sell it anymore, but it's pointless anyway since I never use the zoom... |
2023-07-27 05:38:12.248355587 [INFO] Preparing go2rtc config... |
|
Host: server.local system information
|
I want you to ignore the CPU temps. It's high because ffmpeg is running on another process (immich) via docker to transcode some videos. The issue I have with Frigate not keeping up with frames happens even when the CPU is not stressed. Regardless, if you will insist that this is the issue let me know how to force the frigate container to run as a priority because my security system must always be prioritized above all else. |
Is the disk where recordings are stored on the local machine or are you storing them on a network share? You can turn on debug logging to see the copy times for moving recordings: logger:
logs:
frigate.record: debug If the copy times are fast, then in all other instances this has happened from the CPU load on the machine being too high for Frigate to have the resources to keep up. I do not know of a way to specify priority in HassOS for an addon. Alternatively, you can use Anything you can do to reduce unnecessary processing for frigate will help. You are running detect at a very high resolution which is not necessary for most users, but perhaps you are trying to detect people who are very very far away. If not, using a lower resolution stream or downscaling should help. Make sure you have setup motion masks so it isn't wasting cycles running analysis on things like timestamps. |
It's a SATA disk in the host device. I get this issue even when the CPU is in the low percent of usage. I knew this was going to come up... It may be that, I'm not going to say "it's not that", without data neither of us don't know, so it might be. But it happens even when there's no high CPU utilization. Nick told me to increase shm, he said that was the issue. But it didn't fix, you can see my shm is quite high. I cannot use motion for recording, It's simply not reliable enough and the police said they need an hour before and an hour after any incident and if there is any skipping they cannot accept the evidence. The detect documentation said that the resolution must be native from the camera otherwise the CPU needs to scale the image which consumes more CPU. I noticed this answer in several support tickets, to keep the detect at the original res. Is the issue related to motion detection? Or is the issue related to recording? I can try motion masks but the whole point of this project is smart detection isn't it... I mean nobody can say my system is slow, it's better than most people's main personal computers!! I used to run on a 8GB DDR3 Intel Core i7 3770T processor without GPU (apart from built-in) But that was digressing. I just need support for the issue right now. In terms of the debug logging, before I enable I want to know - why are you asking me to enable debug logging (what specifically do you need to see from it)? Oh and I know I shouldn't expect anyone to assume but it goes without saying I appreciate your time and effort in building this application and supporting others and myself. Big thank you. Never think I am criticizing you. |
This is no longer true, if you use ffmpeg presets for hwaccel (like you are) then the scaling will be done on the GPU instead so it is not a problem.
That is objectively false. You can look at the commits for the upcoming 0.13 release, there have been many multiprocessing improvements for recording management and otherwise. But for now the recommendations are made for the current version, which is mainly to reduce the amount of segments that are kept so the recordings maintainer has less work to do.
The record debug logging will show the amount of time it takes to move the recording segments from cache to the storage drive, which is crucial to understanding where the slow down is. |
Also just wanted to confirm, the |
Other users experiencing this problem have had dozens of cameras. I don't think we have seen anyone else run into this issue with as few cameras as you. |
Hey Nick. What resolution do you suggest me to use then? Remember I'm talking about what we have currently, up to 0.12. I can't say anything about something I have not yet used... so until 0.12 what I said is at least partly applicable. Again please never take my comments as criticism. I am merely saying that I feel priority is being given to features rather than stability. I will enable the debug logging now, thank you. The /db/ is on the SATA drive because I run my container with a bind mount, and the mount is stored in the /home partition which is in a 14TB mirroring RAID. This is my docker CLI:
|
Right and that's what is making me scratch my head. |
For info I run a 3TB WD Red Pro drive for my footage, it's mounted on /cctv |
To make absolutely clear I store my footage in a separate SATA drive (3TB WD Red Pro) and my frigate db is in the main RAID array (14TB). The DB and the frigate config/db are in separate storage drives therefore read/write are separate... |
Looks like your cameras are 4:3 so you could try detect:
width: 1280
height: 960 and see if the objects are detected as you would like. Without seeing the camera images it is difficult to suggest fully because smaller objects that are farther away of course require more resolution to be detected reliably. I run all of my cameras with a height of 720 (and the 4:3 cameras with a height of 960) and have not had any issues detection people far away from the camera.
I would suggest that there were a number of stability improvements in 0.12 as well, but I understand where you're coming from and of course you wouldn't know about the commits in the unreleased version. The to-do list for 0.13 is getting closer so a beta should not be too far away. Another thing that could be tried here given that you require
Great, the values in general will be helpful but it will be mostly helpful to see what the copy times are after the segment cache errors start occurring. |
Cam image: Thanks, that's an empathetic response! I really appreciate it!!! I am doing all I can to try to help you help me, and I know you're a stellar person here with your support. It's really nice when someone recognizes comments and takes it on board, understanding that the intentions are always good. The problem with reducing the bitrate is that it will likely pixellate... this will greatly affect number plate in vehicles especially at night. But the cameras are all feeding directly into a Ubiquiti USW-24-PoE switch which isn't even fully utilized, each camera obviously using their own cable and going into their specific switch port. There is no "in between", it connects directly to my main switch, which the server is also directly connected into. My cameras are outputting to MediaMTX (rtsp-simple-server) and I input the proxied stream into Frigate. If I input the MediaMTX proxied stream for any of the cams into VLC I get a "perfect" stream. I put perfect into quotes because the audio, for some reason, is slow-mo. You know when you record someone speaking and then you slow-mo and the person sounds like their voice is deeper and slower? This is happening and I don't know why. The video itself is real-time, there is a timestamp as you can see above and it's playing back second to second as expected. I'll now post a new log with debug in a separate comment |
|
Based on that camera image I think 1280x960 for detect should work well, you can always verify by watching the debug live view. As far as the debug logs go, it does look like the storage is the problem. The storage is taking a decent amount of time to move segments. Given that you have 4 cameras, there are 4 10-second segments created every 10 seconds. Even if your 4 cameras have the fastest shown times of 3 seconds to move each segment to the storage that is still 12 seconds total to move 1 segment per camera. Which means that your recording storage is not able to keep up with everything that is being written. I definitely think the multiprocessing and other improvements in 0.13 will greatly improve this, but it seems that in your case storage is the limiting factor right now. |
Thanks, having read that, all I can say is that on 0.11 and before these issues weren't really present... Or if they were, I just never noticed. The only time I noticed skipping was when I used a ffmpeg format that included the audio track, but Frigate just does not like it from my cam, so I drop it. I use the generic preset. I suggest... How about we look into my camera output in more detail and see if we can debug what's being sent to Frigate? Do you want a raw output stream? I can send you a link from cloud storage for debug purposes. What do you propose other than the storage? It's kinda hard to accept that a hard drive with over 100MBps write capacity can't keep up with videos that are at most 8MB per file. Even with four cameras it would be around 40MBps at most... that drive is exclusively for Frigate footage. SMART output follows in next comment |
|
Frigate 0.11 did not have these warning logs so there would be no way to know except notice the recording segment wasn't there.
For one it's not constant write it's separate writes for each segment. The timing logs are very simple around the ffmpeg call to optimize the segment metadata and write to storage so there's no room for error in that calculation. Can you show your docker compose? One explanation would be if /tmp/cache isn't using RAM but is using disk instead. |
Odd, definitely seems like the cameras are sending bad data. I'm surprised AAC only has an option for 32000, maybe try PCM restart mediamtx and see what happens in Vic and frigate |
When you say try PCM do you mean G711/G711A? Actually, with AAC the camera UI shows only 16k, not 32. But on VLC it is reported as 32. It's Chinese.... half-assed clearly. Supposed to be Hik clones. The image sensors they use are actually good, the Police has commented on how shar the image looks - except on h265... I have seen a difference. H264 is way sharper on these. I paid £44 per camera. It was a new brand seller on Amazon at the time, when I still used Amazon. So I got it for a bargain. Optical zoom with autofocus... No PTZ sadly, but the zoom allows for more flexible placement. But clearly the firmware is crap. I am sure that it might be possible to find out exactly what is being reported regardless of what the camera says, and manually set ffmpeg/frigate to deal with it accordingly? We will see. I am severely sleep deprived. I caught a 2nd burglary this week. 2 in one week. It's stressful. I really depend on Frigate, and appreciate your efforts. It's not about money. It's about not supporting China. I don't want to buy their NVRs, seems like every NVR is made there. I stopped buying Chinese goods a few years ago. Only as a last resort. Frigate is open, and you guys are doing a great job from the US. I want to see Frigate take off the way Home Assistant did, and I believe it will happen. We need a good, western-based open source Linux NVR system. Keep at it. I'll get back to you. |
Try G711A first would be my suggestion |
All good now. You won't believe... seems like the audio issue was linked to the frame dropping... Once the h265 got enabled and the frames stopped dropping so much, looks like the audio started working.
Once the hard drive is installed, we can then confirm the issue (which I saw added by blake on 0.13 tracker). You said that you use an SSD as cache to speed things up... Actually, I remembered I have a Crucial M4 I am not using, so I am thinking, could you provide me the guide you used to create your cached storage solution with SSD and HDD? I am using Debian 12. I will have a go at this.... this CCTV is being a life saver this month. Two burglaries and a neighbor dispute... |
That should be it
That was added before, when users were seeing this issue running 0.13 after recordings maintainer was moved to its own process. Since then more improvements and optimizations have been added so as long as a users storage can keep up, Frigate should not have any issues.
I run unRAID OS which uses |
I'm currently transferring the data to the new HDD. The audio is working but it's also not working - while there is audio, it seems like it's slowed down, it's not 1x speed but seems like 0.5x speed, so the voice is distorted, sounds like a slow-speaking very low tone voice... like when you use a sound editing app and slow the audio. That's what is happening. So it's pretty useless. Any ideas? I can send you a sample clip if that helps. |
To me it just seems that the cameras audio is bugged quite a bit which is making things difficult. You could try using |
So the audio saga comes to an end. The camera firmware is dumb. It seems to report 16k but sends data at 8k sampling rate anyway. That's why it was "slowed by half". I set the cam to 8k and it's now absolutely clear. So - now I have video AND audio working at full res. If you want me to send you my exact specs to add my brand and model to the documentation please let me know. |
Oh my goodness.... You're not going to like this. New disk installed. WD Purple with 256MB of cache.
it's worse than it was with the 3TB WD Red 9yo drive. And this is with h265 enabled. |
Hmm, that doesn't really make sense why the write would be so much slower than mine with a similar drive. What are the sizes of these segments that are being written? |
As you can see between the post and now, the time has gone down. So there is something happening process-wise we need to find out and eliminate. I don't know what it could be, but this is not acceptable, if it's the OS we need to see what's happening and see if we can tweak or turn on/off some setting. If it's Frigate we need to fix it in Frigate. But what we cannot have is reduced reliability. Regardless of the source Frigate will be seen as the make-or-break application if users install it and notice that footage is skipping... it could drive them away. I know it's more complicated than that but I'd argue a lot of people won't know and won't care to investigate. I hope we can continue through this very long issue and find the root and find a fix for it. |
We have already identified and fixed the processing side of recording maintenance in Frigate for 0.13, which really only comes in to play when the disk transfers are fast but frigate still falls behind. The |
Can you tell me exact what the issue is then? Because I don't think it's that simple. I bought a brand new HDD like you asked me to do. One with even more cache than yours. And my system is no slouch. Do you have a GTX1080 in your system? Do you have a SSD running Debian 12 on it capable of over 5000MB/s reads and writer? Have you got a 13th Gen Intel Processor running frigate? Fast DDR5 Mem, a whole 32GB of it? Please make a comparison with your own system and lets stop the hardware issue line because it ain't it. I am more inclined to think it's software-related. I will draw the line on being a hardware issue now. It's not and I won't spend money on it anymore. We should have debug methods available to find out what's the issue. If Frigate is lacking a more detailed debugging interface now is the time to implement that... Please discuss with Blake and let me know how we should process but any suggestion that it's hardware I won't be receptive to it. You said that you use UNRAID with some unorthodox mixed cache situation - most people don't have that. Most people have hard drives. I bought a hard drive SPECIFICALLY marketed and optimized for NVRs. While I attempt to be polite 100% of the time I do not attempt to be "liked", I know you won't like me for saying this but it's the truth: if Frigate wants to be taken seriously as a NVR solution it needs to work around user's limitations not the other way around. My system is orders of magnitude more than most systems that your application will ever touch - I KNOW THAT. I know that because I read people's posts with their specs and they use far slower machines. It was an expensive machine to build for my purposes, I overprovisioned it so there would be no doubt that it's NOT THE HARDWARE. Frigate needs to consider as a project what is the average device that it will be running on and adapt. It needs to have a robust set of debug interfaces and it needs to more than anything focus on reliability because faulty footage is as good as nothing in criminal proceedings. Understand that my comment is NOT personal - it's targeted at the application. Understand that my comment has GOOD INTENTIONS - I want to see this app be the one thing anyone things of when they think smart NVR. And the way to get there is to make its main function work well. All the detections in the world won't matter if the file is not saved in a way that is usable in court!
We need to investigate this further. I don't know where to start. |
Ps I have dyslexia and I just woke up, there will be typos. |
I think my comment was ambiguous, by hardware issue I was not suggesting the system in general is slow. Users run on intel nucs and don't have this problem that you are seeing. By hardware I meant something with the interface for the HDD, or a host issue by which I meant an issue with the host os / underlying configuration.
A write cache is not unorthodox, it is a feature built in to ZFS, fuse.fs in UnraidOS case, and a feature for other file systems as well. But in any case, I did tests above without using the cache and write speeds directly to the HDD were still much faster than what you are seeing for a 10MB file. |
Exactly - and how many people you know are using ZFS or paying for UNRAID? My system is barebones Debian. It's much more likely to be that way. Nobody is going to be running ZFS on a Raspberry Pi for instance. Let's stop talking about mine vs yours, it's pointless as you've just confirmed after testing, and let's focus on finding the issue - while I might be the only one, we need to be prepared for the possibility that actually, there might be other people with this issue who are not aware or have not bothered to report it. |
I disagree, your HDD has more cache but is otherwise similar to mine and mine has much faster write speeds for a similar mp4 file. That is useful information. What file system do you have on the HDD? Mine uses xfs. |
Well you're only confirming what I said - that it should not be happening because I got the HDD you told me you were using. Only more cache. I use Ext4 which is optimized for small random operations. |
The physical hardware is the same, but there is a large difference in the amount of time to write a 10MB mp4 file. In my opinion this points to an issue outside of Frigate, and support here or Frigate itself does not know the details of the system on the host so it is very difficult to make specific suggestions on what to change. |
I have an idea for something to try, I downloaded this 12 MB sample mp4 from https://download.samplelib.com/mp4/sample-15s.mp4 From within Frigate shell I ran the ffmpeg command to optimize the metadata and move it to my HDD (no cache involved) and the rtime is 0.028 seconds Running the test on the same file should help further reduce variables in the comparison. ffmpeg -benchmark -hide_banner -y -i /media/frigate/exports/sample-15s.mp4 -c copy -movflags +faststart /sandbox/test.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/media/frigate/exports/sample-15s.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf58.43.100
Duration: 00:00:15.65, start: 0.000000, bitrate: 6091 kb/s
Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 6021 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
Metadata:
handler_name : ISO Media file produced by Google Inc. Created on: 08/17/2020.
vendor_id : [0][0][0][0]
Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
Metadata:
handler_name : ISO Media file produced by Google Inc. Created on: 08/17/2020.
vendor_id : [0][0][0][0]
Output #0, mp4, to '/sandbox/test.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf59.27.100
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 6021 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
Metadata:
handler_name : ISO Media file produced by Google Inc. Created on: 08/17/2020.
vendor_id : [0][0][0][0]
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
Metadata:
handler_name : ISO Media file produced by Google Inc. Created on: 08/17/2020.
vendor_id : [0][0][0][0]
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x5628919c1980] Starting second pass: moving the moov atom to the beginning of the file
frame= 464 fps=0.0 q=-1.0 Lsize= 11637kB time=00:00:15.62 bitrate=6100.5kbits/s speed= 568x
video:11379kB audio:240kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.154343%
bench: utime=0.003s stime=0.012s rtime=0.028s
bench: maxrss=26184kB |
I can give you any details you want if you ask me what you want to know |
Here's something interesting I cd into my ~ directory, which is a RAID1 array of 2x 16TB disks in mirror mode. Then I cd into /cctv/test which is the new 4TB drive and ran the same wget (while Frigate container was stopped, of course): If I do at my / which is a NVMe SSD this is the result: Clearly, something is wrong because the /cctv volume is saving much slower, but this is a brand new disk. Could it be the way I mount it on fstab? UUID=b963c953-82d9-4cf7-b6f6-xxxxxxxxxxxxxx /cctv ext4 discard,relatime,noexec,sync 0 2 How are your disks mounted? "discard, relatime, noexec, sync" is how mine is mounted. |
When I run
Other relevant info: |
We come to an end with success - I would highly advise you to put in your documentation that the drive MUST be mounted with ASYNC parameter. The SYNC parameter was causing the issue.
|
This is how it's mounted now, with sub-second copy times:
|
I can confirm I now have a fully working installation with audio and no skips. |
I feel confident about closing this issue now. I want to thank you for your time and patience and thank you and Blake for making this application and for supporting it. You are both top people. |
Describe the problem you are having
Per the title Log says "unable to keep up with recording segments". I don't remember ever getting this on 0.11. I am getting lots of skips in my recordings. The server is more powerful than the one I was using before, but the HDD is the same. The only difference is the Frigate version. What's wrong with my config? I enabled G711 audio in my camera's stream, this is the only difference, but Frigate does not accept it anyway. Even with AAC it doesn't really work and never has. I tried a lot of different ways/presets. Do I have to completely disable audio in the camera to make this work?
Version
0.12.1-367D724
Frigate config file
Relevant log output
FFprobe output from your camera
Frigate stats
Operating system
HassOS
Install method
HassOS Addon
Coral version
USB
Network connection
Wired
Camera make and model
Anpviz 5MP
Any other information that may be helpful
No response
The text was updated successfully, but these errors were encountered: