NerfBaselines is a framework for evaluating and comparing existing NeRF methods. Currently, most official implementations use different dataset loaders, evaluation protocols, and metrics, which renders the comparison of methods difficult. Therefore, this project aims to provide a unified interface for running and evaluating methods on different datasets in a consistent way using the same metrics. But instead of reimplementing the methods, we use the official implementations and wrap them so that they can be run easily using the same interface.

Please visit the project page to see the results of implemented methods on dataset benchmarks.

Start by installing the nerfbaselines pip package on your host system.

pip install nerfbaselines

Now you can use the nerfbaselines cli to interact with NerfBaselines.

WARNING: the default installation only installs the core nerfbaselines package which does not depend on either PyTorch or JAX. However, the LPIPS metric requires PyTorch to be installed and will be disabled otherwise. Similarly, if you install JAX and have a GPU available, the dataloading and evaluation will be faster as some parts of the pipeline will be moved to GPU. Therefore, we recommend installing the extras package by following the Advanced installation section.

The next step is to choose the backend which will be used to install different methods. At the moment there are the following backends implemented:

• docker: Offers good isolation, requires docker to be installed and the user to have access to it (being in the docker user group).
• apptainer: Similar level of isolation as docker, but does not require the user to have privileged access.
• conda (not recommended): Does not require docker/apptainer to be installed, but does not offer the same level of isolation and some methods require additional dependencies to be installed. Also, some methods are not implemented for this backend because they rely on dependencies not found on conda.
• python (not recommended): Will run everything directly in the current environment. Everything needs to be installed in the environment for this backend to work.

The backend can be set as the --backend <backend> argument or using the NERFBASELINES_BACKEND environment variable.

The LPIPS metric requires PyTorch to be installed and will be disabled otherwise. Similarly, if you install JAX and have a GPU available, the dataloading and evaluation will be faster as some parts of the pipeline will be moved to GPU. In this section we describe how to install the packages required for LPIPS and accelerated dataloading. We recommend this as the default installation (unless there is a reason for not installing PyTorch or JAX). Select one of the following configurations:

• CPU-only install

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install jax[cpu]
pip install 'nerfbaselines[extras]'

• CUDA 11.8 install

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install jax[cuda11_pip]
pip install 'nerfbaselines[extras]'

• CUDA 12.1 install

pip install torch torchvision torchaudio
pip install jax[cuda12_pip]
pip install 'nerfbaselines[extras]'

For some datasets, e.g. Mip-NeRF 360 or NerfStudio, the datasets can be downloaded automatically. You can specify the argument --data external:https://dataset/scene during training or download the dataset beforehand by running nerfbaselines download-dataset dataset/scene. Examples:

# Downloads the garden scene to the cache folder.
nerfbaselines download-dataset mipnerf360/garden

# Downloads all nerfstudio scenes to the cache folder.
nerfbaselines download-dataset nerfstudio

# Downloads kithen scene to folder kitchen
nerfbaselines download-dataset mipnerf360/kitchen -o kitchen

To start the training, use the nerfbaselines train --method <method> --data <data> command. Use --help argument to learn about all implemented methods and supported features.

The nerfbaselines render --checkpoint <checkpoint> command can be used to render images from a trained checkpoint. Again, use --help to learn about the arguments.

Given a trained checkpoint, the interactive viewer can be launched as follows:

nerfbaselines viewer --checkpoint <checkpoin> --data <dataset>

Even though the argument --data <dataset> is optional, it is recommended, as the camera poses are used to perform gravity alignment and rescaling for a better viewing experience. Again, you can use the --backend <backend> flag or NS_BACKEND=<backend> environment variable to change the backend.

In this section, we present results of implemented methods on standard benchmark datasets. For detailed results, visit the project page: https://jkulhanek.com/nerfbaselines

Mip-NeRF 360 is a collection of four indoor and five outdoor object-centric scenes. The camera trajectory is an orbit around the object with fixed elevation and radius. The test set takes each n-th frame of the trajectory as test views. Detailed results are available on the project page: https://jkulhanek.com/nerfbaselines/mipnerf360

Blender (nerf-synthetic) is a synthetic dataset used to benchmark NeRF methods. It consists of 8 scenes of an object placed on a white background. Cameras are placed on a semi-sphere around the object. Detailed results are available on the project page: https://jkulhanek.com/nerfbaselines/blender

Nerfstudio Dataset includes 10 in-the-wild captures obtained using either a mobile phone or a mirror-less camera with a fisheye lens. We processed the data using either COLMAP or the Polycam app to obtain camera poses and intrinsic parameters. Detailed results are available on the project page: https://jkulhanek.com/nerfbaselines/nerfstudio

Methods:

• Nerfacto
• Instant-NGP
• Gaussian Splatting
• Mip-Splatting
• Tetra-NeRF
• Mip-NeRF 360
• Zip-NeRF
• CamP
• TensoRF (Blender, LLFF datasets)
• Mip-NeRF
• NeRF

Datasets/features:

• Mip-NeRF 360 dataset
• Blender dataset
• any COLMAP dataset
• any NerfStudio dataset
• automatic dataset download
• interactive viewer
• undistorting images for methods that do not support complex camera models (Gaussian Splatting)
• logging to tensorboard, wandb
• LLFF dataset (only supported in TensoRF now)
• Tanks and Temples
• HDR images support
• RAW images support
• handling large datasets
• loading/creating camera trajectories in the interactive viewer

Contributions are very much welcome. Please open a PR with a dataset/method/feature that you want to contribute. The goal of this project is to slowly expand by implementing more and more methods.

This project is licensed under the MIT license.

A big thanks to the authors of all implemented methods for the great work they have done.