The Triton backend for TensorRT. You can learn more about Triton backends in the backend repo. Ask questions or report problems on the issues page. This backend is designed to run a serialized TensorRT engine models using the TensorRT C++ API.

Where can I ask general questions about Triton and Triton backends? Be sure to read all the information below as well as the general Triton documentation available in the main server repo. If you don't find your answer there you can ask questions on the main Triton issues page.

The command-line options configure properties of the TensorRT backend that are then applied to all models that use the backend.

Instruct TensorRT to consider the requests' inputs with the same name as one contiguous buffer if their memory addresses align with each other. This option should only be enabled if all requests' input tensors are allocated from the same memory region. Default value is false.

Appropriate version of TensorRT must be installed on the system. Check the support matrix to find the correct version of TensorRT to be installed.

$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install ..
$ make install

The following required Triton repositories will be pulled and used in the build. By default the "main" branch/tag will be used for each repo but the listed CMake argument can be used to override.

• triton-inference-server/backend: -DTRITON_BACKEND_REPO_TAG=[tag]
• triton-inference-server/core: -DTRITON_CORE_REPO_TAG=[tag]
• triton-inference-server/common: -DTRITON_COMMON_REPO_TAG=[tag]