• [Feb 2024] GLiDR accepted in CVPR'2024.

The code was tested using pytorch 3.9.18. For topology based installation requirement we use the this fantastic TopologyLayer installation. It mostly works out of the box. There may be some issues while installing Dionysus Drivers for persistence calculation. Pleae refer to the issues at the above repository for quick resolution.

Alternatively you may also follow the below conda environment setup.

git clone https://github.com/prashkmr/GLiDR
cd GLiDR
conda env create -f env.yml
conda activate GLiDR

For training GLiDR we require paired correspondences for static and dynamic scans. These are available for CARLA and ARD-16. For KITTI, we generate the paired correspondence using a novel method that we develop in our paper. See our Supplementary material for precise details.

• [CARLA] - The CARLA dataset consists of 15 correspondence numpy arrays (0..7 for training and the rest for testing) for training and testing the model. We use the first three numpy arrays(0,1,2) to train the model (due to time intensive persistence computation). We further test the model on 8 (8,9...15) numpy arrays. For SLAM we use the 4 CARLA SLAM sequences made available. The data is available here.
• [KITTI] - For KITTI we use the KITTI Odometry sequences for training the model. We transform the 11 sequences into numpy files and make them available here. We use the sequence 8 for testing the model and the rest for training our model For SLAM we use all the Odometry sequences (0 to 10). The static-dynamic corresponding pairs for KITTI are available here.
• [ARD-16] - The ARD-16 dataset consists of 4 numpy arrays (0..3). We use three (0..2) numpy arrays for training our model and the fourth numpy array for testing our model. The data is available here.

Note the dataset structure inside the path you have given in the bash files should look like this:  
**NYUv2**: 
``` bash
nyu
├── nyu_depth_v2
│   ├── official_splits
│   └── sync

KITTI:

kitti
├── KITTI
│   ├── 2011_09_26
│   ├── 2011_09_28
│   ├── 2011_09_29
│   ├── 2011_09_30
│   └── 2011_10_03
└── kitti_gt
    ├── 2011_09_26
    ├── 2011_09_28
    ├── 2011_09_29
    ├── 2011_09_30
    └── 2011_10_03

Please download the pretrained weights from this link and save .ckpt weights inside <repo root>/depth/checkpoints directory.

Also download the v1-5 checkpoint of stable-diffusion and put it in the <repo root>/checkpoints directory. Please create an empty directory if you find that such a path does not exist. Note that this checkpoints folder is different from the one above.

To perform inference on any RGB image or video use the infer_{outdoor,indoor}.sh file. Set the --img_path argument to the image you would to get the depth for and the --video_path to the video from which to produce the depth. In case you only wish to infer on an img or video, simply remove the other argument. Then enter the depth directory by executing cd depth and run:

1. Infer on outdoor scenes: bash infer_outdoor.sh

2. Infer on outdoor scenes: bash infer_indoor.sh

To evaluate the model performance on NYUv2 and KITTI datasets, use the test_{kitti, nyu} file. The trained models are publicly available, download the models using above links. Then, navigate to the depth directory and follow the instructions outlined below:

1. Evaluate on NYUv2 dataset:
   bash test_nyu.sh <path_to_saved_model_of_NYU>

2. Evaluate on KITTI dataset:
   bash test_kitti.sh <path_to_saved_model_of_KITTI>

We trained our models on 32 batch size using 8xNVIDIA A100 GPUs. Inside the train_{kitti,nyu}.sh set the NPROC_PER_NODE variable and --batch_size argument to the desired values as per your system resources. For our method we set them as NPROC_PER_NODE=8 and --batch_size=4 (resulting in a total batch size of 32). Afterwards, navigate to the depth directory by executing cd depth and follow the instructions:

1. Train on NYUv2 dataset:
   bash train_nyu.sh

2. Train on KITTI dataset:
   bash train_kitti.sh

If you have any questions about our code or paper, kindly raise an issue on this repository.

We thank Kartik Anand for assistance with the experiments. Our source code is inspired from VPD and PixelFormer. We thank their authors for publicly releasing the code.

If you find our work useful in your research, please consider citing using:

@article{patni2024ecodepth,
  title={ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation},
  author={Patni, Suraj and Agarwal, Aradhye and Arora, Chetan},
  journal={arXiv preprint arXiv:2403.18807},
  year={2024}
}