The experimental code of "Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives" in ECCV 2022.
Recently, Implicit Neural Representations (INRs) parameterized by neural networks have emerged as a powerful and promising tool to represent all kinds of signals due to its continuous, differentiable properties, showing many superiorities to classical discretized representations. Nevertheless, training of neural networks for INRs only utilizes input-output pairs, and the derivatives of target output with respect to the input which can be accessed in some cases are usually ignored. In this paper, we propose a training paradigm for INRs whose target output is image pixels, to encode image derivatives in addition to image values within the neural network. Specifically, we use finite differences to approximate image derivatives. Further, the neural network activated by ReLUs is poorly suited for representing complex signal's derivatives under the derivative supervision in practice, so the periodic activation function is adopted to get better derivative convergence properties. Lastly, we show how the training paradigm can be leveraged to solve typical INRs problems, such as image regression, inverse rendering, and demonstrate this training paradigm can improve the data-efficiency and generalization capabilities of INRs.
-
Clone this repo
git clone https://github.com/prstrive/Sobolev_training_INRs.git cd Sobolev_training_INRs
-
Install dependencies
Python 3 dependencies (click to expand)
- PyTorch >= 1.10
- torchvision
- ConfigArgParse
- einops
- imageio
- kornia
- matplotlib
- numpy
- opencv_python
- Pillow
- scipy
- tqdm
To setup a conda environment:
conda create -n st_inrs python=3.7 conda activate st_inrs pip install -r requirements.txt
- Create a directory with command:
mkdir data
- Download data:
- Download Set5 for image regression task.
- Download LLFF data from NeRF authors' drive for inverse rendering task.
- Download
gt_bach.wav
andgt_counting.wav
from SIREN authors' drive for audio regression task. Put two WAV files to folderAudio
.
- Create soft links:
ln -s [path to nerf_llff_data] ./data ln -s [path to Set5] ./data ln -s [path to Audio] ./data
cd Experiments/image_regression
python main.py --config [config txt file]
For example, training with value and derivative supervision on Baby with a sine-based model:
python main.py --config configs/baby/val_der/sine.txt
cd Experiments/inverse_rendering
python train.py --config [config txt file]
For example, train with value and derivative supervision on Fern with a ReLU-based MLP:
python train.py --config configs/fern/val_der/relu.txt
After training for 400K iterations, you can find novel view results in logs/fern/val_der/relu/testset_400000
, you can evaluate results with following command and score.txt
will be generated in logs/fern/val_der/relu/testset_400000/score.txt
:
python eval.py --config configs/fern/val_der/relu.txt
cd Experiments/audio_regression
python main.py --config [config txt file]
For example, training with value supervision and on Bach with a sine-based model:
python main.py --config configs/bach/val_sine.txt
If you find our work useful in your research, please cite:
@inproceedings{yuan2022sobolev,
title={Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives},
author={Wentao Yuan and Qingtian Zhu and Xiangyue Liu and Yikang Ding and Haotian Zhang and Chi Zhang},
year={2022},
booktitle={ECCV},