TensorRTx aims to implement popular deep learning networks with tensorrt network definition APIs. As we know, tensorrt has builtin parsers, including caffeparser, uffparser, onnxparser, etc. But when we use these parsers, we often run into some "unsupported operations or layers" problems, especially some state-of-the-art models are using new type of layers.

So why don't we just skip all parsers? Instead, build the entire network directly using the tensorrt network definition API, which is not that complicated.

I wrote this project to get familiar with tensorrt API, and also to share and learn from the community.

Get the trained models from pytorch, mxnet or tensorflow, etc. Export the weights to .wts file. Then load weights in tensorrt, define network and do inference. Some pytorch models can be found in my repo pytorchx, the remaining are from popular open-source repo.

• 26 Oct 2022. ausk: YoloP(You Only Look Once for Panopitic Driving Perception).
• 19 Sep 2022. QIANXUNZDL123 and lindsayshuo: YOLOv7.
• 7 Sep 2022. xiang-wuu: YOLOv5 v6.2 classification models.
• 19 Aug 2022. Dominic and sbmalik: Yolov3-tiny and Arcface support TRT8.
• 6 Jul 2022. xiang-wuu: SuperPoint - Self-Supervised Interest Point Detection and Description, vSLAM related.
• 26 May 2022. triple-Mu: YOLOv5 python script with CUDA Python API.
• 23 May 2022. yhpark: Real-ESRGAN, Practical Algorithms for General Image/Video Restoration.
• 19 May 2022. vjsrinivas: YOLOv3 TRT8 support and Python script.
• 15 Mar 2022. sky_hole: Swin Transformer - Semantic Segmentation.
• 19 Oct 2021. liuqi123123 added cuda preprossing for yolov5, preprocessing + inference is 3x faster when batchsize=8.
• 18 Oct 2021. xupengao: YOLOv5 updated to v6.0, supporting n/s/m/l/x/n6/s6/m6/l6/x6.
• 31 Aug 2021. FamousDirector: update retinaface to support TensorRT 8.0.
• 27 Aug 2021. HaiyangPeng: add a python wrapper for hrnet segmentation.
• 1 Jul 2021. freedenS: DE⫶TR: End-to-End Object Detection with Transformers. First Transformer model!
• 10 Jun 2021. upczww: EfficientNet b0-b8 and l2.

• Install the dependencies.
• A guide for quickly getting started, taking lenet5 as a demo.
• The .wts file content format
• Frequently Asked Questions (FAQ)
• Migrating from TensorRT 4 to 7
• How to implement multi-GPU processing, taking YOLOv4 as example
• Check if Your GPU support FP16/INT8
• How to Compile and Run on Windows
• Deploy YOLOv4 with Triton Inference Server
• From pytorch to trt step by step, hrnet as example(Chinese)

1. TensorRT 7.x
2. TensorRT 8.x(Some of the models support 8.x)

Each folder has a readme inside, which explains how to run the models inside.

Following models are implemented.

The .wts files can be downloaded from model zoo for quick evaluation. But it is recommended to convert .wts from pytorch/mxnet/tensorflow model, so that you can retrain your own model.

GoogleDrive | BaiduPan pwd: uvv2

Some tricky operations encountered in these models, already solved, but might have better solutions.

Help wanted, if you got speed results, please add an issue or PR.

Any contributions, questions and discussions are welcomed, contact me by following info.

E-mail: [email protected]

WeChat ID: wangxinyu0375 (可加我微信进tensorrtx交流群，备注：tensorrtx)