Any tips importing a custom TensorRT Model for Nvidia Detection? #9161

meriley · 2024-01-01T16:29:08Z

meriley
Jan 1, 2024

I want to preface by saying, I'm really excited for the upcoming Frigate+. Unfortunately, what I want to detect right now does not fit in its current model. Crossing my fingers for some custom model support as it gets more mature!

I have a camera setup inside a bearded dragon enclosure, and I want to create a custom model to detect the Lizards movements to different parts of his enclosure. I'm able to train a Tensor Flow model and get detection independent of Frigate, but I'm struggling to get it into a format that Frigate will accept. I've tried a few methods with out much luck;

GPU TensorRT

Create a Tensor Flow Saved Model using Teachable Machine (Root Cause 1 in no success I'm sure)
Convert to Onyx

python -m tf2onnx.convert --saved-model /frigate/models/rex_tf_savedmodel/model.savedmodel --output rex.onnx

Convert to TensorRT

docker run --gpus all -v $PWD:/workspace nvcr.io/nvidia/tensorrt:22.07-py3 /bin/bash \
  -c "cd /workspace && trtexec --onnx=rex.onnx --saveEngine=rex.trt"

Config:

detectors:
  tensorrt:
    type: tensorrt
    device: 0

model:
  path: /trt-models/rex.trt
  input_tensor: nhwc
  input_pixel_format: rgb
  width: 224
  height: 224

Output:

2024-01-01 09:11:01.256411627  [2024-01-01 09:11:01] frigate.detectors.plugins.tensorrt ERROR   : bad dims of binding sequential_3: (1, 2)
2024-01-01 09:11:01.256887902  Traceback (most recent call last):
2024-01-01 09:11:01.256919101    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 228, in __init__
2024-01-01 09:11:01.256921007      ) = self._allocate_buffers()
2024-01-01 09:11:01.256922850    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 125, in _allocate_buffers
2024-01-01 09:11:01.256924164      raise ValueError(
2024-01-01 09:11:01.256934534  ValueError: bad dims of binding sequential_3: (1, 2)
2024-01-01 09:11:01.256935611  
2024-01-01 09:11:01.256937066  The above exception was the direct cause of the following exception:
2024-01-01 09:11:01.256938070  
2024-01-01 09:11:01.256944864  Traceback (most recent call last):
2024-01-01 09:11:01.256968030    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-01-01 09:11:01.256969201      self.run()
2024-01-01 09:11:01.256970651    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-01-01 09:11:01.256971987      self._target(*self._args, **self._kwargs)
2024-01-01 09:11:01.256973422    File "/opt/frigate/frigate/object_detection.py", line 98, in run_detector
2024-01-01 09:11:01.256989784      object_detector = LocalObjectDetector(detector_config=detector_config)
2024-01-01 09:11:01.256999388    File "/opt/frigate/frigate/object_detection.py", line 52, in __init__
2024-01-01 09:11:01.257000804      self.detect_api = create_detector(detector_config)
2024-01-01 09:11:01.257002735    File "/opt/frigate/frigate/detectors/__init__.py", line 24, in create_detector
2024-01-01 09:11:01.257004044      return api(detector_config)
2024-01-01 09:11:01.257005762    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 231, in __init__
2024-01-01 09:11:01.257007112      raise RuntimeError("fail to allocate CUDA resources") from e
2024-01-01 09:11:01.257008373  RuntimeError: fail to allocate CUDA resources
2024-01-01 09:11:01.257054125  Exception ignored in: <function TensorRtDetector.__del__ at 0x15114ade5a60>
2024-01-01 09:11:01.257055332  Traceback (most recent call last):
2024-01-01 09:11:01.257062791    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 238, in __del__
2024-01-01 09:11:01.257176228      if self.outputs is not None:
2024-01-01 09:11:01.257185215  AttributeError: 'TensorRtDetector' object has no attribute 'outputs'

CPU and Tensorflow Lite

Since I wasnt having much luck with TRT I decided to try a CPU based model

Config

detectors:
  cpu1:
    type: cpu
    model:
      path: /tflite-models/rex.tflite

model:
  labelmap_path: /tflite-models/rex_labels.txt
  width: 224
  height: 224
  input_tensor: "nhwc"
  input_pixel_format: "bgr"

Output:

2024-01-01 09:15:56.637029942  Process detector:cpu1:
2024-01-01 09:15:56.637762242  Traceback (most recent call last):
2024-01-01 09:15:56.637782488    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-01-01 09:15:56.637784562      self.run()
2024-01-01 09:15:56.637786340    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-01-01 09:15:56.637791982      self._target(*self._args, **self._kwargs)
2024-01-01 09:15:56.637801732    File "/opt/frigate/frigate/object_detection.py", line 121, in run_detector
2024-01-01 09:15:56.637803744      detections = object_detector.detect_raw(input_frame)
2024-01-01 09:15:56.637805606    File "/opt/frigate/frigate/object_detection.py", line 71, in detect_raw
2024-01-01 09:15:56.637807134      return self.detect_api.detect_raw(tensor_input=tensor_input)
2024-01-01 09:15:56.637809031    File "/opt/frigate/frigate/detectors/plugins/cpu_tfl.py", line 44, in detect_raw
2024-01-01 09:15:56.637822859      class_ids = self.interpreter.tensor(self.tensor_output_details[1]["index"])()[0]
2024-01-01 09:15:56.637832609  IndexError: list index out of range

Thoughts

The root of the issue is that I am clearly not a AI / Deep Learning expert. I've just gotten sucked into this rabbit hole because I want to tinker and know when my Lizard eats!

Ive generated the models using Teachable Machine, because its barrier of entry was lowest. This is likely the main issue since the models that work with Frigate are YOLO based. I've not seen any discussions on anyone successfully training a model on custom data set outside of the YOLO labels.

I'd love to hear any successes, and any documents / tips you may have to help me model my lizard in a way that Frigate can work with the model. I am going to try and use this method: https://blog.paperspace.com/yolov7/ and report back when I have time!

Answered by meriley

Jan 2, 2024

I believe I finally got something working! Using Yolov8 even!
Its not refined by any means, but it compiles and plugs into frigate with no errors.

Create A New Project With Roboflow
Upload Images for detection and label accordingly
Download Model Locally (Don't Bother with Roboflow Training)
Train the model (I believe the training writes a TON of data to the image. So you need to delete the image after every run because its ~14GB after training. I haven't investigated where it writes so I haven't exposed a volume for clean up yet.)

echo '#!/bin/bash

NAME=rex

# Remove Old Generated Model
rm -rf /workspace/models/$NAME

# Train Model
yolo train data=/workspace/$NAME/data.yaml project=/w…

View full answer

NickM-27 · 2024-01-01T17:23:34Z

NickM-27
Jan 1, 2024
Collaborator Sponsor

The coral / cpu models are based on mobiledet by default. @NateMeyer / @blakeblackshear might be able to offer more specific advice, there are other users on Frigate as well that use custom trained models as well.

0 replies

NateMeyer · 2024-01-01T17:53:07Z

NateMeyer
Jan 1, 2024

The shape of the tensors in your model is different from what the TensorRT detector is expecting. One option you could do is copy the detector from frigate/detectors/plugins/tensorrt.py and modify it to correctly parse the results from your model into the results array that gets returned to Frigate. This could be mapped to the detectors/plugins directory with a bind mount in docker so you don't have to rebuild the whole image.

Make sure you pass in a valid label map that includes a row for every possible class that the model will return. Looks like your tflite model returned a class that didn't exist in your label map.

0 replies

meriley · 2024-01-02T05:29:59Z

meriley
Jan 2, 2024
Author

I believe I finally got something working! Using Yolov8 even!
Its not refined by any means, but it compiles and plugs into frigate with no errors.

Create A New Project With Roboflow
Upload Images for detection and label accordingly
Download Model Locally (Don't Bother with Roboflow Training)
Train the model (I believe the training writes a TON of data to the image. So you need to delete the image after every run because its ~14GB after training. I haven't investigated where it writes so I haven't exposed a volume for clean up yet.)

echo '#!/bin/bash

NAME=rex

# Remove Old Generated Model
rm -rf /workspace/models/$NAME

# Train Model
yolo train data=/workspace/$NAME/data.yaml project=/workspace/models name="$NAME" batch=2 model=yolov8n.pt epochs=50 lr0=0.001 imgsz=512

# Export Model
yolo export project=/workspace/output name="$NAME" imgsz=512 model=yolov8n.pt format=onnx
yolo export project=/workspace/output name="$NAME" imgsz=512 model=yolov8n.pt format=engine

# Manually Move output files to the disk volume
mv /usr/src/ultralytics/yolov8n.engine /workspace/yolov8n.engine
mv /usr/src/ultralytics/yolov8n.onnx /workspace/yolov8n.onnx' > train.sh

chmod +x train.sh

docker run \
  --gpus all \
  --shm-size 8G \
  --rm -it \
  -v $PWD:/workspace \
  -v $PWD/train.sh:/train.sh \
  ultralytics/ultralytics \
  /train.sh

Convert Onnx file to TensorRT (nvidia/tensorrt:23.03-py3 is important since Frigate and Ultralytics use that image)

docker run --gpus all -v $PWD:/workspace nvcr.io/nvidia/tensorrt:23.03-py3 /bin/bash \
  -c "cd /workspace && trtexec --onnx=yolov8n.onnx --saveEngine=rex.trt"

Setup Frigate to use new Model

model:
  path: /trt-models/rex.trt
  input_tensor: nchw
  input_pixel_format: rgb
  width: 512
  height: 512

Verify Logs

2024-01-01 22:33:54.312494327  [2024-01-01 22:33:54] frigate.detectors.plugins.tensorrt INFO    : Loaded engine size: 18 MiB
2024-01-01 22:33:54.435138465  [2024-01-01 22:33:54] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +7, GPU +12, now: CPU 140, GPU 80 (MiB)
2024-01-01 22:33:54.438895033  [2024-01-01 22:33:54] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +19, now: CPU 0, GPU 19 (MiB)
2024-01-01 22:33:54.479169387  [2024-01-01 22:33:54] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 122, GPU 80 (MiB)
2024-01-01 22:33:54.482056048  [2024-01-01 22:33:54] frigate.detectors.plugins.tensorrt INFO    : [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +12, now: CPU 0, GPU 31 (MiB)
2024-01-01 22:33:54.482130888  [2024-01-01 22:33:54] frigate.detectors.plugins.tensorrt WARNING : The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.

I don't know how well it works in Frigate yet, but I believe that is a model / training issue, not a Frigate implementation issue.

The only thing I don't fully understand is why the TensorRT File Generated by Ultralytics was not just plug and play. I needed to pass a onnx through trtexec to generate a usable TRT. When using the Ultralytics created TensorRT I see:

2024-01-01 22:29:22.588537063  [2024-01-01 22:29:22] frigate.detectors.plugins.tensorrt ERROR   : 1: [stdArchiveReader.cpp::StdArchiveReader::32] Error Code 1: Serialization (Serialization assertion magicTagRead == kMAGIC_TAG failed.Magic tag does not match)
2024-01-01 22:29:22.600957852  [2024-01-01 22:29:22] frigate.detectors.plugins.tensorrt ERROR   : 4: [runtime.cpp::deserializeCudaEngine::66] Error Code 4: Internal Error (Engine deserialization failed.)
2024-01-01 22:29:22.602971681  Process detector:tensorrt:
2024-01-01 22:29:22.605085676  Traceback (most recent call last):
2024-01-01 22:29:22.605114529    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-01-01 22:29:22.605116627      self.run()
2024-01-01 22:29:22.605118573    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-01-01 22:29:22.605120546      self._target(*self._args, **self._kwargs)
2024-01-01 22:29:22.605132032    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
2024-01-01 22:29:22.605133555      object_detector = LocalObjectDetector(detector_config=detector_config)
2024-01-01 22:29:22.605135369    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
2024-01-01 22:29:22.605136929      self.detect_api = create_detector(detector_config)
2024-01-01 22:29:22.605138643    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2024-01-01 22:29:22.605140588      return api(detector_config)
2024-01-01 22:29:22.605142069    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 221, in __init__
2024-01-01 22:29:22.605157611      self.input_shape = self._get_input_shape()
2024-01-01 22:29:22.605159160    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 93, in _get_input_shape
2024-01-01 22:29:22.605183950      binding = self.engine[0]
2024-01-01 22:29:22.605196333  TypeError: 'NoneType' object is not subscriptable
2024-01-01 22:29:22.605197816  Exception ignored in: <function TensorRtDetector.__del__ at 0x14c818b7e280>
2024-01-01 22:29:22.605199271  Traceback (most recent call last):
2024-01-01 22:29:22.605200520    File "/opt/frigate/frigate/detectors/plugins/tensorrt.py", line 239, in __del__
2024-01-01 22:29:22.605281770      if self.outputs is not None:
2024-01-01 22:29:22.605292616  AttributeError: 'TensorRtDetector' object has no attribute 'outputs'

4 replies

alexyao2015 Feb 17, 2024

It seems like yolov8 appends some metadata to the model file so you have to load it using the following.

with open(model_path, "rb") as f, trt.Runtime(self.trt_logger) as runtime:
    meta_len = int.from_bytes(
        f.read(4), byteorder="little"
    )  # read metadata length
    metadata = json.loads(f.read(meta_len).decode("utf-8"))  # read metadata
    model = runtime.deserialize_cuda_engine(f.read())  # read engine
    return model

Unfortunately, I'm still hitting some issues decoding the detection results after this which I still need to understand.

alexyao2015 Feb 19, 2024

Applying the code here to trt_outputs along with the fixes to the model loading, I was able to get yolov8 working properly with frigate and tensorrt!

return self.yolov8_postprocess(self.input_shape, trt_outputs[0].reshape((1, -1, 2100)))

alexyao2015 Feb 22, 2024

I have an open pr with the code for this.
#9954

meriley Feb 23, 2024
Author

I guess an update, while I did initially get a model to be accepted by Frigate, it didn't really work. Tons of false positives in seemingly inactive areas. When I have time I'll try your changes!

Did you generate your TRT with the instructions I posted, or directly from ultralytics?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Any tips importing a custom TensorRT Model for Nvidia Detection? #9161

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 4 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Any tips importing a custom TensorRT Model for Nvidia Detection? #9161

meriley Jan 1, 2024

Replies: 3 comments · 4 replies

NickM-27 Jan 1, 2024 Collaborator Sponsor

NateMeyer Jan 1, 2024

meriley Jan 2, 2024 Author

alexyao2015 Feb 17, 2024

alexyao2015 Feb 19, 2024

alexyao2015 Feb 22, 2024

meriley Feb 23, 2024 Author

meriley
Jan 1, 2024

Replies: 3 comments 4 replies

NickM-27
Jan 1, 2024
Collaborator Sponsor

NateMeyer
Jan 1, 2024

meriley
Jan 2, 2024
Author

meriley Feb 23, 2024
Author