Example of radar-camera depth completion: (a) raw radar depth, (b) enhanced radar depth, (c) final predicted depth.
rc-pda/
data/
nuscenes/
annotations/
maps/
samples/
sweeps/
v1.0-trainval/
lib/
scripts/
external/
panoptic-deeplab/
RAFT/
- Create a conda environment called pda
conda create -n pda python=3.6- Install required packages
pip install -r requirements.txt- Download nuScenes dataset (Full dataset (v1.0)) into data/nuscenes/
- Clone external repos Panoptic-DeepLab and RAFT into external/
1. Data preparation
cd scripts
# 1) split data
python split_trainval.py
# 2) extract images for flow computation
python prepare_flow_im.py
# 3) compute image flow from im1 to im2
python cal_flow.py
# 4) compute camera intrinsic matrix and transformation from cam1 to cam2
python cal_cam_matrix.py
# 5) transform image flow to normalized expression (u2,v2)
python cal_im_flow2uv.py
# 6) compute vehicle semantic segmentation
python semantic_seg.py
# 7) compute dense ground truth (depth1, u2, v2) and low height mask
python cal_gt.py
# 8) compute merged radar (5 frames)
python cal_radar.py
# 9) create .h5 dataset file
python gen_h5_file3.py 2. Estimate radar camera association
python train_pda.py # train
python test_pda.py # testDownload pre-trained weights
3. Generate enhanced radar depth (RC-PDA)
python cal_mer.py4. Train depth completion by using the enhanced depth
- Depth completion scheme 1 (Using depths and RGB as input channels)
python train_depth.py # train
python test_depth.py # testDownload pre-trained weights
- Depth completion scheme 2 (Multi-Scale Guided Cascade Hourglass Network)
python train_depth_hg.py # train
python test_depth_hg.py # testDownload pre-trained weights.
@InProceedings{Long_2021_CVPR,
author = {Long, Yunfei and Morris, Daniel and Liu, Xiaoming and Castro, Marcos and Chakravarty, Punarjay and Narayanan, Praveen},
title = {Radar-Camera Pixel Depth Association for Depth Completion},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2021},
pages = {12507-12516}
}