"Even with these dark eyes, a gift of the dark night, I go to seek the shining light." --Gu Cheng 1956-1993

Please check our videos (Bilibili).

The sister repo Neural Light Simulator for light-camera calibration is here.

Installation generally follows vanilla Gaussian Splatting installation.

git clone [email protected]:tyz1030/darkgs.git --recursive

or

git clone https://github.com/tyz1030/darkgs.git --recursive

Conda environment setup

conda env create --file environment.yml
conda activate darkgs

Also need to install lietorch

pip install git+https://github.com/princeton-vl/lietorch.git

Please find our example data on Google Drive and DropBox.

Please put your RAW images subfolder named "raw". To make COLMAP less struggle, I gamma-curved/manually increased the brightness of the raw images for feature extraction and matching. These corrected images are put in "input" subfolder. We only use "raw" images to build DarkGS.

python3 convert.py -s <path to your own dataset>

If you are using your own light-camera setup, please calibrate your system using neural light simulator. Then put model_parameters.pth in the root directory.

Train

python train.py -s <path to example dataset>

Visualize with SIRB viewer:

./SIBR_remoteGaussian_app

Then you will be able to steer your light cone by pressing "JKLI" on the keyboard.\

SIBR_gaussianViewer_app is not compatible with this repo. Please try the following:

python3 viz_chkpt.py -s data/lab1/ -m output/<xxxxxx-xxx> --start_checkpoint output/<xxxxxx-xxx>/chkpnt30000.pth

then in another terminal

./SIBR_remoteGaussian_app

Meanwhile, there is no one-true-solution to relighting.
I have ADHD symptoms, so please forgive my slow progress.
One quick hack through is when running viz_chkpt.py, uncomment line 129 in scene/lighting.py. And you will also need to brighten, white balance and gamma correct the final render results to make it look good otherwise it is in blue-greenish RAW format.

This work is picked up by IROS 2024 as oral presentation! Arxiv

@INPROCEEDINGS{zhang2024darkgs,
  author={Tianyi Zhang and Kaining Huang and Weiming Zhi and Matthew Johnson-Roberson},
  booktitle={2024 International Conference on Intelligent Robots and Systems (IROS)}, 
  title={DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic Exploration in the Dark}, 
  year={2024}}

• This work is supported by NOAA.
• Copyright 2024 Tianyi Zhang, Carnegie Mellon University. All rights reserved.