[Question]: Using the new review items for notifications

[ofcedwards]

What is your question:

I've been testing out the new frigate/review mqtt topic to generate notifications in Home Assistant and have found a few things that don't quite work as expected (I'm currently running 0.14.0-bccffe6).

1. Using FRIGATE_URL:5000/review?id=REVIEW_ID as a link provides a different experience than clicking on a review item from the Frigate UI. Review items clicked from the UI always start playback earlier than if the same review item is viewed by manually navigating to it through the url. I believe this is related to a const called REVIEW_PADDING that looks to be in play when a user clicks a review item. From what I can tell, this const backs up the start time by two seconds. Looking through the code, I don't see anything similar for when review items are viewed by url. This two second difference has been repeatable for me across every review item I've tested (both navigating by url to view the item and clicking on it in the ui). The two seconds is also long enough that it makes the difference between seeing the event actually start and seeing it in progress (or if a car is going by, seeing the car disappear as soon as playback starts). I'd love to use this feature because having all events merged into one review item is excellent in 0.14.0, but having to skip back at the start of every video has made me switch back to getting the event clip through the api.
2. Relating to the new way all detected objects are merged together. I've set Home Assistant to only notify on "alerts" from Frigate, but alerts will include any other object detected along with what I have set (people in a certain zone). I have no issue with this because it's much better to have one section of video to playback to capture all events, but I'm finding it difficult to grab a snapshot of the object that caused the alert. Object detections are now provided in mqtt as list, but pulling from the first item in the list does not consistently provide the desired thumbnail (a person, in my case). At times, I get a thumbnail of the person triggering the alert, but other times I get a thumbnail of a car that was walked in front of or a car driving by on the street. The thumbnails in the review section of the ui seem to constantly be what I'd want, but the .webp images to do not show up in notifications via Home Assistant on my iPhone (17.5.1). If there's some way to sort out the event id of what caused the detection from detection to alert, I'd be very grateful to know. Also, if anyone is getting .webp images to work on an iPhone, I'd love to know how.
3. Occasionally, notifications will be significantly delayed as the detection in Frigate is still in progress. The worst I've experienced was approximately a two minute delay between when the person was detected to when an alert was sent. It seemed that Frigate struggled to decide that it should be an alert (the person was detected when the appeared, but the clip was in the detection page, not the alert page). I believe this had to do with the detection starting with a car outside the zone, then entering the zone, then a person getting out of it (which should finally change it to an alert). I can only guess that somewhere in that interaction, Frigate got bogged down in determining the classification and only decided when it finalized the review item. I run Frigate in docker on unraid with a coral, so there was definitely not a hardware issue.

Any guidance on these three items would be greatly appreciated. I'm loving the new ui in 0.14.0 and the changes to the way events are reviewed has made checking over footage much more efficient. Frigate has gone from something I'd mostly use in Home Assistant to a standalone "app" I have saved to the home screen on my phone. I'd love to use some of these new changes in the way I receive notifications, but the combination of all three items has me sticking with the old method for now.

[NickM-27]

1. I agree, will adjust
2. I would suggest using the preview gif as the image in the notification instead of a single object's snapshot
3. I've not seen this. There is no real way for it to bog down, perhaps there was just not an update for some reason. Will see if others report the same

[ofcedwards]

I was on dev-c9d253a at the time of that event, which I believe should have included the fix?

[ofcedwards]

The update issue appears to be fixed. I had the same circumstances today (same car, person, and positioning) and the event was changed from detection to alert as expected. However, the mqtt payloads are still mirrored. This is on dev-6978140. I'm not sure if this section of the review maintainer may be the cause:

        prev_data = segment.get_data(ended=False)

        if frame is not None:
            segment.update_frame(camera_config, frame, objects)

        new_data = segment.get_data(ended=False)

This is starting at line 199 and is under the update_segment function. It appears that prev_data and new_data are being defined as the same thing. The function called in the middle (update_frame) appears to only relate to the image that's displayed for the segment. I could be completely wrong here because I'm not super familiar with python, but to me it appears that both variables would be equal and would result in the same output.

[NickM-27]

yeah, that was mainly to account for the image path but it is correct that some parts are set before that happens, will fix

[ofcedwards]

Fixed! Tested this out on the new dev image and it's showing all the expected changes now (no zone to zone, detection to alert, list of event ids changes length).

Thank you for all the help through this! 0.14 has been a huge upgrade and I'm really excited to see where frigate goes in the future. You guys are really killing it.