如何使用rtmo的SDK版onnx模型进行推理

[lzk020813]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmpose).

Environment

Python' : '3.8.18
CUDA available' : True
PyTorch' : '1.10.0

Reproduces the problem - code sample

我想使用rtmo的SDK版onnx模型进行推理，用mmdeploy中的pose_detection.py进行测试，代码如下

import argparse
import cv2
import numpy as np
from mmdeploy_runtime import PoseDetector


def parse_args():
    parser = argparse.ArgumentParser(
        description='show how to use sdk python api')
    parser.add_argument('device_name', help='name of device, cuda or cpu')
    parser.add_argument(
        'model_path',
        help='path of mmdeploy SDK model dumped by model converter')
    parser.add_argument('image_path', help='path of an image')
    parser.add_argument(
        '--bbox',
        default=None,
        nargs='+',
        type=int,
        help='bounding box of an object in format (x, y, w, h)')
    args = parser.parse_args()
    return args


def main():
    args = parse_args()

    img = cv2.imread(args.image_path)

    detector = PoseDetector(
        model_path=args.model_path, device_name=args.device_name, device_id=0)

    if args.bbox is None:
        result = detector(img)
    else:
        # converter (x, y, w, h) -> (left, top, right, bottom)
        print(args.bbox)
        bbox = np.array(args.bbox, dtype=int)
        bbox[2:] += bbox[:2]
        result = detector(img, bbox)
    print(result)

    _, point_num, _ = result.shape
    points = result[:, :, :2].reshape(point_num, 2)
    for [x, y] in points.astype(int):
        cv2.circle(img, (x, y), 1, (0, 255, 0), 2)

    cv2.imwrite('output_pose.png', img)


if __name__ == '__main__':
    main()

Reproduces the problem - command or script

运行命令 python pose_detection.py cpu d:/rtmpose-ort/rtmo d:/1.jpg 进行测试，其中d:/rtmpose-ort/rtmo为sdk目录，但好像无法实现

Reproduces the problem - error message

出现以下信息

loading mmdeploy_trt_net.dll ...
failed to load library mmdeploy_trt_net.dll
loading mmdeploy_ort_net.dll ...
[2024-01-06 15:41:32.149] [mmdeploy] [info] [model.cpp:35] [DirectoryModel] Load model: "d:/rtmpose-ort/rtmo"
[2024-01-06 15:41:32.149] [mmdeploy] [error] [compose.cpp:37] Unable to find Transform creator: BottomupResize. Available transforms: [("CenterCrop", 0), ("Collect", 0), ("Compose", 0), ("DefaultFormatBund
le", 0), ("FormatShape", 0), ("ImageToTensor", 0), ("LetterResize", 0), ("Lift", 0), ("LoadImageFromFile", 0), ("Normalize", 0), ("Pad", 0), ("RescaleToHeight", 0), ("Resize", 0), ("ResizeOCR", 0), ("Short
ScaleAspectJitter", 0), ("TenCrop", 0), ("ThreeCrop", 0), ("TopDownAffine", 0), ("TopDownGetBboxCenterScale", 0)]
[2024-01-06 15:41:32.149] [mmdeploy] [error] [task.cpp:99] error parsing config: {
  "context": {
    "device": "<any>",
    "model": "<any>",
    "stream": "<any>"
  },
  "input": [
    "img"
  ],
  "module": "Transform",
  "name": "Preprocess",
  "output": [
    "prep_output"
  ],
  "transforms": [
    {
      "type": "LoadImageFromFile"
    },
    {
      "input_size": [
        640,
        640
      ],
      "pad_val": [
        114,
        114,
        114
      ],
      "type": "BottomupResize"
    },
    {
      "mean": [
        0,
        0,
        0
      ],
      "std": [
        1,
        1,
        1
      ],
      "to_rgb": false,
      "type": "Normalize"
    },
    {
      "keys": [
        "img"
      ],
      "type": "ImageToTensor"
    },
    {
      "keys": [
        "img"
      ],
      "meta_keys": [
        "img_shape",
        "pad_shape",
        "ori_shape",
        "img_norm_cfg",
        "scale_factor",
        "bbox_score",
        "center",
        "scale"
      ],
      "type": "Collect"
    }
  ],
  "type": "Task"
}
[2024-01-06 15:41:32.474] [mmdeploy] [error] [common.h:50] Could not found entry 'UNKNOWN' in mmpose. Available components: [("DeepposeRegressionHeadDecode", 0), ("SimCCLabelDecode", 0), ("TopdownHeatmapBa
seHeadDecode", 0), ("TopdownHeatmapMSMUHeadDecode", 0), ("TopdownHeatmapMultiStageHeadDecode", 0), ("TopdownHeatmapSimpleHeadDecode", 0), ("ViPNASHeatmapSimpleHeadDecode", 0)]
[2024-01-06 15:41:32.475] [mmdeploy] [error] [task.cpp:99] error parsing config: {
  "component": "UNKNOWN",
  "context": {
    "device": "<any>",
    "model": "<any>",
    "stream": "<any>"
  },
  "input": [
    "prep_output",
    "infer_output"
  ],
  "module": "mmpose",
  "name": "postprocess",
  "output": [
    "post_output"
  ],
  "params": {
    "flip_test": false,
    "input_size": [
      640,
      640
    ],
    "nms_thr": 0.65,
    "score_thr": 0.1,
    "type": "YOLOXPoseAnnotationProcessor"
  },
  "type": "Task"
}

Additional information

之前没有用sdk模型处理过one-stage的算法，请问我该怎样才能用rtmo的SDK版onnx模型进行推理呢，十分感谢

[Ben-Louis]

可以尝试下 rtmlib

[danigarciaoca]

Same problem on my side, when converting RTMO to TRT and trying to perform inference with MMDeploy

[sxj731533730]

ubuntu@ubuntu:~/mmdeploy$ python3 tools/deploy.py configs/mmpose/pose-detection_rtmo_tensorrt-fp16_dynamic-640x640.py /home/ubuntu/mmpose/configs/body_2d_keypoint/rtmo/body7/rtmo-t_8xb32-600e_body7-416x416.py https://download.openmmlab.com/mmpose/v1/projects/rtmo/rtmo-t_8xb32-600e_body7-416x416-f48f75cb_20231219.pth /home/ubuntu/mmpose/tests/data/coco/000000000785.jpg --work-dir onnx_dir --dump-info --show --device cuda:0

face the problem

RuntimeError: Unsupported: ONNX export of transpose for tensor of unknown rank.
01/30 08:46:38 - mmengine - ERROR - /home/ubuntu/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - mmdeploy.apis.pytorch2onnx.torch2onnx with Call id: 0 failed. exit.

########################################################

ubuntu@ubuntu:~/mmdeploy$ python3 tools/deploy.py configs/mmpose/pose-detection_rtmo_onnxruntime_dynamic.py /home/ubuntu/mmpose/configs/body_2d_keypoint/rtmo/body7/rtmo-t_8xb32-600e_body7-416x416.py https://download.openmmlab.com/mmpose/v1/projects/rtmo/rtmo-t_8xb32-600e_body7-416x416-f48f75cb_20231219.pth /home/ubuntu/mmpose/tests/data/coco/000000000785.jpg --work-dir onnx_dir --dump-info --show --device cuda:0

face the problem

RuntimeError: Exporting the operator linalg_norm to ONNX opset version 11 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.
01/30 08:35:59 - mmengine - ERROR - /home/ubuntu/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - mmdeploy.apis.pytorch2onnx.torch2onnx with Call id: 0 failed. exit.

[luuuyi]

ubuntu@ubuntu:~/mmdeploy$ python3 tools/deploy.py configs/mmpose/pose-detection_rtmo_tensorrt-fp16_dynamic-640x640.py /home/ubuntu/mmpose/configs/body_2d_keypoint/rtmo/body7/rtmo-t_8xb32-600e_body7-416x416.py https://download.openmmlab.com/mmpose/v1/projects/rtmo/rtmo-t_8xb32-600e_body7-416x416-f48f75cb_20231219.pth /home/ubuntu/mmpose/tests/data/coco/000000000785.jpg --work-dir onnx_dir --dump-info --show --device cuda:0

face the problem

RuntimeError: Unsupported: ONNX export of transpose for tensor of unknown rank. 01/30 08:46:38 - mmengine - ERROR - /home/ubuntu/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - mmdeploy.apis.pytorch2onnx.torch2onnx with Call id: 0 failed. exit.

########################################################

ubuntu@ubuntu:~/mmdeploy$ python3 tools/deploy.py configs/mmpose/pose-detection_rtmo_onnxruntime_dynamic.py /home/ubuntu/mmpose/configs/body_2d_keypoint/rtmo/body7/rtmo-t_8xb32-600e_body7-416x416.py https://download.openmmlab.com/mmpose/v1/projects/rtmo/rtmo-t_8xb32-600e_body7-416x416-f48f75cb_20231219.pth /home/ubuntu/mmpose/tests/data/coco/000000000785.jpg --work-dir onnx_dir --dump-info --show --device cuda:0

face the problem

RuntimeError: Exporting the operator linalg_norm to ONNX opset version 11 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub. 01/30 08:35:59 - mmengine - ERROR - /home/ubuntu/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - mmdeploy.apis.pytorch2onnx.torch2onnx with Call id: 0 failed. exit.

also face this problem

[luuuyi]

可以尝试下 rtmlib

能问下你的pytorch是什么版本的么，转rtmo的op_set是多少

[Ben-Louis]

@luuuyi

[zephyrzhu1998]

@lzk020813 你好，我也遇到了这个问题，请问你后来解决了吗？