-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Support]: Occasional 2 minute delay in Frigate shutdown/restart #8227
Comments
One possibility: if Frigate hasn't shutdown within 30s, could something like https://docs.python.org/3.11/library/faulthandler.html#faulthandler.dump_traceback_later be used to dump a full stack trace? |
I have seen this too. |
Another case today on 0.13.0-beta5 / 0.13.0-AA93D4F
|
I think I managed to catch this using pystack:
Not sure if that helps much. |
I've updated my pystack script with |
looks like there are a couple python sub processes processes open, that additional info will likely be helpful |
Had a reoccurence, pystack output this time with Unfortunately other than showing that the hanging code seems to be Python C code that's trying to run a finalizer I'm not sure if it's much more helpful.
Output of
|
Still occurring in 0.14 dev builds so will have another go at getting some more debug data. |
I'm seeing this in 0.14 too. Not sure if it's a delay or what, but it sits and waits for a while to close the threads, and then finally restarts. For me it's now more-often-than-not. |
Describe the problem you are having
Occasionally, it seems one or more of my cameras can block Frigate from shutting down cleanly. Looking at the logs, I can see shutdown entries for each camera "Terminating the existing ffmpeg process..." followed by "Waiting for ffmpeg to exit gracefully..." except for one camera, which never logs either message.
Nothing happens for 2 minutes, then Frigate is killed, which looks to be based on the timeout in
docker/main/rootfs/etc/s6-overlay/s6-rc.d/frigate/timeout-kill
.I've attached a full log below - to note:
revotech3 logs a "Terminating..." message, but never logs the second "Waiting..." one, suggesting it might be hanging or never reaching the
communicate()
call herefrigate/frigate/video.py
Line 129 in cd35481
revotech1 doesn't log any message at all, suggesting it's not attempting to stop the process.
I can't readily reproduce this, otherwise I'd try and dive in the container and get a stack trace or
ps -ef
dump while it's hanging to see which processes are still around.If this is not straightforward to investigate further, I'm happy to send a PR to add some debug logging (I tend to run in debug anyway) to log out the status of the
Version
0.13.0-DCAFCC1
Frigate config file
n/a
Relevant log output
FFprobe output from your camera
Frigate stats
No response
Operating system
Debian
Install method
Docker Compose
Coral version
USB
Network connection
Wired
Camera make and model
various
Any other information that may be helpful
For comparison, here is a "healthy" exit which takes a few seconds.
The text was updated successfully, but these errors were encountered: