This repository is the official implementation of A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving.
| Name | Supported Versions |
|---|---|
| Ubuntu | 18.04, 20.04 |
| Python | 3.7 ,3.8 |
| CUDA | 10.1 ,10.2, 11.0 |
| Cudnn | 7.6.5 , 8.0.1 |
| PyTorch | 1.5 , 1.5.1, 1.6 |
To install requirements virtualenv and virtualenvwrapper should be available on the target machine.
Virtual Environment Creation:
# Clone repo
git clone https://github.com/asharakeh/pod_compare.git
cd pod_compare
# Create python virtual env
mkvirtualenv pod_compare
# Add library path to virtual env
add2virtualenv src
# Install requirements
cat requirements.txt | xargs -n 1 -L 1 pip install
Download the Berkeley Deep Drive (BDD) Object Detection Dataset here. The BDD
dataset should have the following structure:
└── BDD_DATASET_ROOT
├── info
| └── 100k
| ├── train
| └── val
├── labels
└── images
├── 10K
└── 100K
├── test
├── train
└── val
Download the KITTI Object Detection Dataset here. The KITTI
dataset should have the following structure:
└── KITTI_DATASET_ROOT
├── object
├── training <-- 7481 train data
| ├── image_2
| ├── calib
| └── label_2
└── testing <-- 7580 test data
├── image_2
└── calib
Download the Lyft Object Detection Dataset here. The Lyft
dataset needs to be converted to KITTI format first using the official Lyft dataset API.
The Lyft dataset should have the following structure:
└── LYFT_DATASET_ROOT
└── training
├── image_2
└── label_2
For all three datasets, labels need to be converted to COCO format. To do so, run the following:
python src/core/datasets convert_bdd_to_coco.py --dataset-dir /path/to/bdd/dataset/root
python src/core/datasets convert_kitti_to_coco.py --dataset-dir /path/to/kitti/dataset/root
python src/core/datasets convert_lyft_to_coco.py --dataset-dir /path/to/lyft/dataset/root
If the script to convert BDD labels to COCO format does not work, please use these pre-converted labels.
To train the model(s) in the paper, run this command:
python src/train_net.py
--num-gpus 2
--dataset-dir /path/to/bdd/dataset/root
--config-file BDD-Detection/retinanet/name_of_config.yaml
--random-seed xx
--resume
For running inference and evaluation of a model, run the following code:
python src/apply_net.py --dataset-dir /path/to/test/dataset/root --test-dataset test_dataset_name --config-file BDD-Detection/retinanet/name_of_config.yaml --inference-config Inference/name_of_inference_config.yaml
--test-dataset can be one of bdd_val, kitti_val, or lyft_val. --dataset-dir corresponds to the root directory of the dataset used.
Evaluation code will run inference on the test dataset and then will generate mAP, Negative Log Likelihood, Calibration Error, and Minimum Uncertainty Error results. If only evaluation of metrics is required,
add --eval-only to the above code snippet.
We provide a list of config combinations that generate the architectures used in our paper:
| Method Name | Config File | Inference Config File | Pretrained Model |
|---|---|---|---|
| Baseline RetinaNet | retinanet_R_50_FPN_1x.yaml | standard_nms.yaml | retinanet_R_50_FPN_1x.pth |
| Loss Attenuation | retinanet_R_50_FPN_1x_reg_cls_var.yaml | standard_nms.yaml | retinanet_R_50_FPN_1x_reg_cls_var.pth |
| Loss Attenuation + Dropout | retinanet_R_50_FPN_1x_reg_cls_var_dropout.yaml | mc_dropout_ensembles_pre_nms.yaml | retinanet_R_50_FPN_1x_reg_cls_var_dropout.pth |
| BayesOD | retinanet_R_50_FPN_1x_reg_cls_var.yaml | bayes_od.yaml | retinanet_R_50_FPN_1x_reg_cls_var.pth |
| BayesOD + Dropout | retinanet_R_50_FPN_1x_reg_cls_var_dropout.yaml | bayes_od_mc_dropout.yaml | retinanet_R_50_FPN_1x_reg_cls_var_dropout.pth |
| Pre-NMS Ensembles | retinanet_R_50_FPN_1x_reg_cls_var.yaml | ensembles_pre_nms.yaml | - |
| Post-NMS Ensembles | retinanet_R_50_FPN_1x_reg_cls_var.yaml | ensembles_post_nms.yaml | - |
| Black Box | retinanet_R_50_FPN_1x_dropout.yaml | mc_dropout_ensembles_post_nms.yaml | retinanet_R_50_FPN_1x_dropout.pth |
| Output Redundancy | retinanet_R_50_FPN_1x.yaml | anchor_statistics.yaml | retinanet_R_50_FPN_1x.pth |
Ensemble methods require multiple independent training runs using different random seeds.
To do so, run the training code while adding random-seed xx. We test with 5 runs using seed values of 0, 1000, 2000, 3000, and 4000 in our paper.
For evaluating with PDQ, please use the official PDQ code for COCO data available here.
If you use this code, please cite our paper:
@misc{feng2020review,
title={A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving},
author={Di Feng and Ali Harakeh and Steven Waslander and Klaus Dietmayer},
year={2020},
eprint={2011.10671},
archivePrefix={arXiv},
primaryClass={cs.CV}}
This code is released under the Apache 2.0 License.