This work addresses the problem of point cloud registration using deep neural networks. We propose an approach to predict the alignment between two point clouds with overlapping data content, but displaced origins. Such point clouds originate, for example, from consecutive measurements of a LiDAR mounted on a moving platform. The main difficulty in deep registration of raw point clouds is the fusion of template and source point cloud. Our proposed architecture applies flow embedding to tackle this problem, which generates features that describe the motion of each template point. These features are then used to predict the alignment in an end-to-end fashion without extracting explicit point correspondences between both input clouds. We rely on the KITTI odometry and ModelNet40 datasets for evaluating our method on various point distributions. Our approach achieves state-of-the-art accuracy and the lowest run-time of the compared methods.

Authors: Markus Horn, Nico Engel, Vasileios Belagiannis, Michael Buchholz, Klaus Dietmayer
ArXiv: https://arxiv.org/abs/2007.11255

@inproceedings{horn2020deepclr,
    author={Horn, Markus and Engel, Nico and Belagiannis, Vasileios and Buchholz, Michael and Dietmayer, Klaus},
    booktitle={IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)},
    title={DeepCLR: Correspondence-Less Architecture for Deep End-to-End Point Cloud Registration},
    year={2020},
    pages={1--7},
    doi={10.1109/ITSC45102.2020.9294279}
}

For questions or support please email Markus Horn.

[[TOC]]

The recommended way for using DeepCLR is from within a Docker container. We provide two images: a full image with all dependencies and DeepCLR installed deepclr and an image with dependencies only deepclr-deps. Only the NVIDIA drivers, Docker (https://docs.docker.com/get-docker/), and nvidia-docker (https://github.com/NVIDIA/nvidia-docker) have to be installed on your host system for using these images.

The docker/run_docker.sh script starts a new container in interactive mode, using your current user within the container and forwarding all required environment variables. You can use your current user and mount your home directory with the --user option or mount other directories with -m DIR. Further options are printed with ./docker/run_docker.sh --help.

If you want to pull and launch the full image, simply use

./docker/run_docker.sh --user -m /datasets

In case you want to use the sources from the repository and load the image with only dependencies, add the --deps option. In this case, the script automatically prepends the DeepCLR directory to your PYTHONPATH within the container.

Alternatively, you can also build the images locally using docker/build_docker.sh and run the containers using docker/run_docker.sh with the --local option.

Since our GPU cluster runs on singularity, we also provide docker/docker2singularity.sh for converting a docker image to a singularity image and docker/python_singularity.sh for running python scripts within a singularity container.

Call the following commands from the root directory of this repository to install DeepCLR along with all necessary dependencies. We recommend using a virtualenv for this.

We assume that CUDA 11.1 is already installed. The dependencies torch, torchvision and torch-cluster are installed for this specific version. If you are using another CUDA version, you can change this according to the documentation at https://pytorch.org/ and https://github.com/rusty1s/pytorch_cluster.

# System requirements
sudo apt update
sudo apt install build-essential cmake ghostscript gnuplot libcap-dev libglib2.0-0 libgl1-mesa-glx libusb-1.0-0 libgsl-dev texlive-extra-utils unzip

# (Optional) Create and source virtual environment
python -m venv venv
source venv/bin/activate

# Wheel
python -m pip install wheel

# (Optional) Python dependencies for specific CUDA version
python -m pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html
python -m pip install torch-cluster==1.5.9 -f https://pytorch-geometric.com/whl/torch-1.8.0+cu111.html

# Python dependencies
python -m pip install -r requirements.txt

# Prepare, build and install external dependencies
./extern/prepare.sh
./extern/install.sh

# Install DeepCLR in editable mode
python -m pip install -e .

The script docker/run_tensorboard.sh starts TensorBoard within a separate Docker container and forwards all arguments.

./docker/run_tensorboard.sh ~/models --port 16006

Since we use Tensorpack DataFlow and LMDB for data loading, the original datasets have to be converted first before training. Furthermore, some scripts and configs rely on the environment variables KITTI_PATH, MODELNET40_PATH and MODEL_PATH.

Create a new directory for KITTI Odometry and store this directory in the environment variable KITTI_PATH. Download the velodyne laser data, calibration files and ground truth poses from https://www.cvlibs.net/datasets/kitti/eval_odometry.php and extract them into a new directory ${KITTI_PATH}/original. You should have now the poses at ${KITTI_PATH}/original/poses and the sequences at ${KITTI_PATH}/original/sequences.

Convert the data and create noisy test data using our scripts (this may take a while):

./scripts/converter/kitti_odometry.py
./scripts/converter/kitti_pairs.py
./scripts/converter/kitti_pairs_test.py

Create a new directory for the ModelNet40 and store this directory in the environment variable MODELNET40_PATH. Download the models from https://shapenet.cs.stanford.edu/media/modelnet40_normal_resampled.zip (GitHub: https://github.com/charlesq34/pointnet2) and extract them into a new directory ${MODELNET40_PATH}/original. You should now have the file list at ${MODELNET40_PATH}/original/filelist.txt.

Convert the data and create noisy test data using our scripts (this may take a while):

./scripts/converter/modelnet40.py
./scripts/converter/modelnet40_test.py

If you want to use our pretrained models, set the environment variable MODEL_PATH to the model directory of this repository. Don't forget to run Git LFS for pulling the models.

Otherwise, if you want to train your own models, you can set MODEL_PATH to any path you like, for example ~/models.

This section gives a short overview of the available scripts for training, prediction and evaluation. For more detailed information you can call the respective script with the -h option.

Train model with training configuration. Configs, logs, checkpoints and tensorboard data are stored in a new subdirectory of the MODEL_PATH directory.

./scripts/training.py ./configs/training/kitti_00-06.yaml

Run model inference for a scenario. Results are stored in a new subdirectory of the output directory.

./scripts/inference.py ./configs/scenarios/kitti_07-10.yaml kitti_00-06 ~/scenarios

You can also use scripts/run_inference.sh in order to run the inference with the pretrained models on multiple scenarios.

Run ICP registration for a scenario. Results are stored in a new subdirectory of the output directory.

./scripts/icp.py ./configs/scenarios/kitti_07-10.yaml ICP_PO2PO ~/scenarios

You can also use scripts/run_icp.sh in order to run multiple ICP variants on multiple scenarios.

Evaluate a single inference / ICP run or multiple runs of a given scenario. Results are stored in a subdirectory evaluation of the input directory.

./scripts/evaluation.py ~/scenarios --scenario kitti_07-10

Export inference / ICP results as KITTI poses, compatible with the KITTI Odometry Devkit. Poses are stored in a subdirectory kitti of the input directory.

./scripts/export_kitti_poses.py ~/scenarios/20210101_120000_ICP_PO2PO_kitti_07-10

A slightly modified version of the KITTI Odometry Devkit is also installed in the Docker image. Results are stored in a subdirectory of the predicted poses.

python -m kitti_devkit ${KITTI_PATH}/original/poses ~/scenarios/20210101_120000_ICP_PO2PO_kitti_07-10/kitti

Test inference time for an untrained model.

./scripts/timing.py ./configs/training/kitti_00-06.yaml

The scripts/converter directory contains the preprocessing scripts already mentioned in Datasets and Environment Variables.

The scripts/examples directory contains examples for simple inference on original KITTI and ModelNet40 data.

The scripts/visualization directory contains scripts for showing the output of various processing steps and modules.

The scripts/paper directory contains the scripts for generating the tables and plots used in the DeepCLR paper.

The documentation is built using Sphinx along with sphinx-apidoc:

cd doc
make autogen
make html

After building, the documentation index is located at doc/build/html/index.html.

Unittests are executed with pytest. The code can be checked by flake8 and type hints are checked by mypy.

We usually use single quotes for strings and double quotes only for natural language, such as logs, error messages, or documentation.

The code is tested within our Docker and on Ubuntu 18.04 with Python 3.8.0 and CUDA 11.1.

PyTorch: https://pytorch.org/
Tensorpack DataFlow: https://tensorpack.readthedocs.io/en/latest/tutorial/dataflow.html
PointNet++ for PyTorch: https://github.com/sshaoshuai/Pointnet2.PyTorch
Generalized ICP: https://github.com/avsegal/gicp
KITTI Odometry Devkit: https://www.cvlibs.net/datasets/kitti/eval_odometry.php
PyBind11: https://github.com/pybind/pybind11