Code and Data for our CVPR 2022 paper Compressive Single-Photon 3D Cameras.

Project Page: https://pages.cs.wisc.edu/~felipe/project-pages/2022-compressive-histograms/

• compressive-spad-lidar-cvpr22
  • Getting Started
    • Step 1. Setup Python environment
    • Step 2. Setup data paths
    • Step 3. Download the data
  • Reproducing Flash LiDAR Results
  • Reproducing Scan Data Results with the Real-world captured Data
  • Reproducing Simulated Isometric Compression Results and Mean Absolute Error Results
  • Coding Schemes Evaluated In CVPR 2022 Paper
  • Visualization Scripts
  • Notes on Naming Convention
    • Coding Schemes
    • Parameters
  • Reference

Before running any of the scripts in this folder, please follow the following steps.

Create the conda envirionment (csphenv) from the environment.yml:

    conda env create -f environment.yml

Open io_dirpaths.json and set the dirpaths where you want data to be downloaded, results, and data to be saved.

Note: We recommend leaving the default values, which will store everything under the ./data folder.

The main dirpath that can be set (or left as default) is

• data_base_dirpath: This is the main dirpath where the data, results, and images will be saved. All the other dirpaths below will be specified with respect to this dirpath.

The other folder paths here are all relative to data_base_dirpath:

• results_base_dirpath: Dirpath containing most data and figures generated by the scripts
• results_data: Dirpath where the data generated by scripts is stored
• paper_results_dirpath: Dirpath where plotting scripts will output plots to.
• transient_images_dirpath: Dirpath to transient rendered histogram images data. The downloaded data in step 3 is stored in this path.
• rgb_images_dirpath: Dirpath to rgb rendered images for the transient images. The downloaded data in step 3 is stored in this path.
• depth_images_dirpath: Dirpath to ground truth depth images for the transient images. The downloaded data in step 3 is stored in this path.
• scan_data_base_dirpath: Dirpath where the pre-processed scan histogram data is stored. The downloaded data in step 3 is stored in this path.

You should be able to run scripts/download_data.py from the top-level folder:

python scripts/download_data.py

The data will be download inside the ./data folder. If you want to change the folder where the data is downloaded edit the parameter data_base_dirpath inside io_dirpaths.json.

You can run the eval_coding_flash_lidar_scene_batch.sh script. In the script you can modify the desired sbr, nphotons (photon counts), K, to use in the simulation. You can also comment in and out the blocks of code that will simulate different coding schemes. The results will be saved under results/results_data/eval_coding_flash_lidar. Running the script for all coding schemes may take a few minutes.

To visualize the results you can use the script plotting_scripts/plot_flash_lidar_sim_results.py. Make sure the parameters here are set to the same simulation parameters when you ran eval_coding_flash_lidar_scene_batch.sh.

To perform individual flash illumination simulations you can run the eval_coding_flash_lidar_scene.py script with the specified parameters. Some sample run commands are at the top of the scipt.

You can run the scan_data_scripts/process_hist_img.py script. Please refer to file file for a description of what the script is doing.

NOTE: Running all coding schemes for many different K values might take hours to days. In particular, TimestampCoding is very slow and takes a while to complete.

You can run the eval_coding_gauss_mu_batch.sh script. In the script you can modify the range of sbr, nphotons (photon counts) levels. Also for each coding scheme you can vary the K values that are used, to use in the simulation. You can also comment in and out the blocks of code that will simulate different coding schemes. The results will be saved under results/results_data/final_coding_gauss_mu_est.

The batch script simply runs eval_coding_gauss_mu_est.py over and over again with different hyper parameter configurations.

The implementation of the coding schemes used in the paper are implemented as individual classes and can be found under tof-lib/toflib/coding.py.

The following classes have a one-to-one correspondence to the coding schemes described in the main paper:

1. GatedCoding: This class corresponds to Coarse Histograms coding scheme
2. TruncatedFourierCoding: This class corresponds to Truncated Fourier coding scheme.
3. PSeriesFourierCoding: This class corresponds to Gray-based Fourier coding scheme. This coding scheme samples frequencies from the Fourier matrix by doubling the frequency that is sampled. Once it cannot double the frequency anymore, it reverts back to TruncatedFourierCoding and samples the remaining frequencies from lowest to highest
4. GrayCoding: This class corresponds to Continuous Gray coding scheme. This coding scheme is exactly the same as Gray coding when K == log2(N). For all other K values the Gray codes are linearly interpolated. Note that this scheme is only valid for K <= log2(N). For a coding scheme that uses approximately binary codes and supports K > log2(N), see PSeriesGray below.
5. IdentityCoding: This class corresponds to Full-resolution Histograms where no compression is applied.
6. TimestampCoding: This class corresponds to Timestamp Transfer approach. This is not necessarily a coding scheme. This approach simply truncated the number of timestamps used to generate the histograms

Furthermore, the following classes have a ont-to-one correspondence with the coding schemes described in the supplementary document:

1. PSeriesGrayCoding: This class corresponds to Fourier-based Gray coding scheme. This coding scheme is similar to PSeriesFourierCoding but uses binarized codes. When K <= log2(N), this coding scheme is the same as Gray
2. GatedFourier-F-1: This class corresponds to short-time fourier coding scheme. This coding scheme first defines a GatedCoding, and then for each gate it defines a fixed number of frequencies to be sampled within that gate. See coding.py:GatedFourierCoding for implementation details.

The plotting_scripts folder has the different scripts used to generate most of the raw figures shown in the paper. For instance running: python plotting_scripts/plot_example_irfs.py will create the IRF plots in the paper.

These scripts will usually save the output images under results/raw_figures.

The scripts that plot the result figures require running the scan data script, flash illumination simulations, and MDE monte carlo simulations to create the results data used by the script.

1. PSeriesFourier == Gray-based Fourier
2. PSeriesGray == Fourier-based Gray
3. Gated == Coarse Histogram
4. GatedFourier-F-1 == Short-time Fourier
5. Identity == Full-Res Histogram

• K or n_codes: Number of rows (coding functions) in coding matrix
• nt or n_tbins: Number of time bins in uncompressed histogram
• nr or n_rows: Number of rows
• nc or n_cols: Number of cols
• rec-zncc-irf, rec-ncc-irf, rec-linear-irf: Algorithms used to compute the look-up table used to estimate depths. ZNCC and NCC depth decoding. linear is only used for coarse histograms. It is not exactly ZNCC but it is faster and performs a bit better because it estimates the depth in the middle of the time bin (ZNCC estimates the depth at the beginning of time bin).

If you find the code and data in this repository useful please cite:

@InProceedings{Gutierrez-Barragan_2022_CVPR,
    author    = {Gutierrez-Barragan, Felipe and Ingle, Atul and Seets, Trevor and Gupta, Mohit and Velten, Andreas},
    title     = {Compressive Single-Photon 3D Cameras},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {17854-17864}
}

Moreover, the raw data used in this paper was derived from two other works:

• If you use the real experimental data, please reference the following projects:

  • https://github.com/felipegb94/WISC-SinglePhoton3DData
  • https://wisionlab.com/project/spad-lidar/

• If you use the rendered transient image data used to simulate the flash lidar scenario, please reference the following:

  • https://pages.cs.wisc.edu/~felipe/project-pages/2021-itof2dtof/
  • Felipe Gutierrez-Barragan, Huaijin Chen, Mohit Gupta, Andreas Velten, Jinwei Gu, "iToF2dToF: A Robust and Flexible Representation for Data-driven Time-of-Flight Imaging". IEEE Transactions on Computational Imaging, 2021