PyTorch implementation of the paper:

2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds.

Minhao Li, Zheng Qin, Zhirui Gao, Renjiao Yi, Chengyang Zhu, Yulan Guo, and Kai Xu.

The commonly adopted detect-then-match approach to registration finds difficulties in the cross-modality cases due to the incompatible keypoint detection and inconsistent feature description. We propose, 2D3D-MATR, a detection-free method for accurate and robust registration between images and point clouds. Our method adopts a coarse-to-fine pipeline where it first computes coarse correspondences between downsampled patches of the input image and the point cloud and then extends them to form dense correspondences between pixels and points within the patch region. The coarse-level patch matching is based on transformer which jointly learns global contextual constraints with self-attention and cross-modality correlations with cross-attention. To resolve the scale ambiguity in patch matching, we construct a multi-scale pyramid for each image patch and learn to find for each point patch the best matching image patch at a proper resolution level. Extensive experiments on two public benchmarks demonstrate that 2D3D-MATR outperforms the previous state-of-the-art P2-Net by around $20$ percentage points on inlier ratio and over $10$ points on registration recall.

2023.10.08: Code and pretrained models on 7Scenes and RGB-D Scenes V2 released.

2023.08.10: Paper is available at arXiv.

2023.07.14: This work is accepted by ICCV 2023.

Please use the following command for installation.

# It is recommended to create a new environment
conda create -n matr2d3d python==3.8
conda activate matr2d3d

# 2. Install vision3d following https://github.com/qinzheng93/vision3d

The code has been tested on Python 3.8, PyTorch 1.13.1, Ubuntu 22.04, GCC 11.3 and CUDA 11.7, but it should work with other configurations.

We provide pre-trained weights from BaiduYun(extraction code: 34ks). Please download the latest weights and put them in weights directory.

The dataset can be downloaded from BaiduYun(extraction code: m7mc). The data should be organized as follows:

--data--7Scenes--metadata
              |--data--chess
                    |--fire
                    |--heads
                    |--office
                    |--pumpkin
                    |--redkitchen
                    |--stairs

The code for 7Scenes is in experiments/2d3dmatr.7scenes.stage4.level3.stage1. Use the following command for training.

CUDA_VISIBLE_DEVICES=0 python trainval.py

Use the following command for testing.

CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH

EPOCH is the epoch id.

We also provide pretrained weights in weights, use the following command to test the pretrained weights.

CUDA_VISIBLE_DEVICES=0 python test.py --checkpoint=/path/to/2D3DMATR/weights/2d3dmatr-7scenes.pth
CUDA_VISIBLE_DEVICES=0 python eval.py --test_epoch=-1

The dataset can be downloaded from BaiduYun(extraction code: 2dc7). The data should be organized as follows:

--data--RGBDScenesV2--metadata
                   |--data--rgbd-scenes-v2-scene_01
                         |--...
                         |--rgbd-scenes-v2-scene_14

The code for RGB-D Scenes V2 is in experiments/2d3dmatr.rgbdv2.stage4.level3.stage1. Use the following command for training.

CUDA_VISIBLE_DEVICES=0 python trainval.py

Use the following command for testing.

CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH

EPOCH is the epoch id.

We also provide pretrained weights in weights, use the following command to test the pretrained weights.

CUDA_VISIBLE_DEVICES=0 python test.py --checkpoint=/path/to/2D3DMATR/weights/2d3dmatr-rgbdv2.pth
CUDA_VISIBLE_DEVICES=0 python eval.py --test_epoch=-1

We evaluate GeoTransformer on the 7Scenes and RGB-D Scenes V2 benchmarks.

NOTE: the results could be a little different due to the randomness of RANSAC-PnP.

@inproceedings{li20232d3d,
  title={2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds},
  author={Li, Minhao and Qin, Zheng and Gao, Zhirui and Yi, Renjiao and Zhu, Chenyang and Guo, Yulan and Xu, Kai},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={14128--14138},
  year={2023}
}

• D2-Net
• D3Feat
• RPMNet
• vision3d
• GeoTransformer