[Detector Support]: Frigate 0.14.0 Openvino with YOLONAS problems

[joakim5937x]

Describe the problem you are having

Hi,

Trying to get yolonas model to work with frigate 0.14.0. And having some problems. I know this is not officially supported but have someone gotten this to work?

Tried converting the model using this code:

import openvino as ov
import torch
from super_gradients.training import models
from super_gradients.common.object_names import Models




# loading an image for testing YOLO-NAS Pose model.
model = models.get(Models.YOLO_NAS_S,
                   pretrained_weights="coco")




# prepare dummy input data
input_data = torch.rand(1, 3, 640, 640)


# converting Pose model to Openvino
ov_model = ov.convert_model(model, example_input=input_data)


# save model to OpenVINO IR for later use
ov.save_model(ov_model, 'model.xml')


# Compile and infer with OpenVINO:
compiled_model = ov.compile_model('model.xml')


# run inference
result = compiled_model(input_data)
print(result)

this gives error:

frigate  | 2024-05-29 21:04:39.188457371    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 36, in __init__
frigate  | 2024-05-29 21:04:39.188458815      logger.info(f"Model Input Shape: {self.interpreter.input(0).shape}")
frigate  | 2024-05-29 21:04:39.188459941  ValueError: get_shape was called on a descriptor::Tensor with dynamic shape

I also tried this:

from super_gradients.training import models
from super_gradients.common.object_names import Models

model = models.get(Models.YOLO_NAS_M, pretrained_weights="coco")

model.eval()
model.prep_model_for_conversion(input_size=[1, 3, 640, 640])
model.export("yolo_nas_m.onnx", postprocessing=None, preprocessing=None)

mo --input_model yolo_nas_m.onnx --model_name yolo_nas_m -s 255 --reverse_input_channels --compress_to_fp16 --input_shape [1,3,640,640] --output_dir content/yolo_nas_m

but this gives error:

frigate  | 2024-05-29 21:34:30.708920519  Traceback (most recent call last):
frigate  | 2024-05-29 21:34:30.708921767    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
frigate  | 2024-05-29 21:34:30.708922740      self.run()
frigate  | 2024-05-29 21:34:30.708923864    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
frigate  | 2024-05-29 21:34:30.708924897      self._target(*self._args, **self._kwargs)
frigate  | 2024-05-29 21:34:30.708943484    File "/opt/frigate/frigate/object_detection.py", line 127, in run_detector
frigate  | 2024-05-29 21:34:30.708944811      outputs[connection_id]["np"][:] = detections[:]
frigate  | 2024-05-29 21:34:30.708946229  TypeError: 'NoneType' object is not subscriptable

Thanks from a ML noob!

Version

0.14.0

Frigate config file

N/A

docker-compose file or Docker CLI command

N/A

Relevant log output

N/A

Operating system

Debian

Install method

Docker Compose

Coral version

Other

Any other information that may be helpful

No response

[NickM-27]

CC @blakeblackshear

[jsapede]

built it according to
https://deci.ai/blog/how-to-optimize-yolo-nas-pose-for-inference-with-openvino/
on a google collab and it seems to work !

simply change the model name to "yolo_nas_s" for exaple and withdraw the part with the picture test and it converts ang generates bin/xml

[jsapede]

tried this morning 0.14beta2 with these models build with collab but doesnt seem to work

[blakeblackshear]

There is a draft PR up for this: #11645

[igiannakas]

It appears that the merged playbook still doesn’t work I think?

[NickM-27]

that is not a breaking error, the notebook still works

[igiannakas]

ah I feel stupid haha! thank you!!