VoGE: A Differentiable Volume Renderer using Neural Gaussian Ellipsoids for Analysis-by-Synthesis [ICLR 2023]
VoGE is a differentiable rendering library conducts differentiable rendering of Gaussian ellipsoids. The rendering process fully follows ray tracing volume densities. VoGE library provide PyTorch APIs for easy usage. For the structure, we refer to PyTorch3D. For the math and details, refer to our VoGE paper.
The VoGE library is written in PyTorch, with some components implemented in CUDA for improved performance. Those components also have CPU only substitutions written in PyTorch (will be release soon). A CUDA support GPU with PyTorch-GPU is suggest (currently required) to use VoGE.
- Linux or Windows
- Python 3.8 or 3.9
- PyTorch 1.10
- PyTorch3D 0.6
- Numpy 1.20
- CUDA 10.2 or 11.3
We only list versions of the required packages that have been tested, other version of the described library might also usable.
Runing the following code to compile and install VoGE library:
git clone https://github.com/Angtian/VoGE.git
cd VoGE
python setup.py install
or
pip install git+https://github.com/Angtian/VoGE.git
Once successfully install VoGE, it should be able to include in your python:
import VoGE
 |
 |
|---|---|
| Single-viewed Texture Extraction | Shape and Color Fitting using VoGE |
 |
 |
|---|---|
| Change Lighting for Rendering Bunny | Rendering PointCloud |
 |
 |
|---|---|
| Using VoGE | Baseline: SoftRas |
| Single-viewed Occlusion Reasoning -> | for Multi Objects |
 |
... |
|---|---|
| Efficient Cuboid via Optimization | More |
The in wild object pose estimation experiment using NeMo pipeline will be released in the NeMo project page.
 |
 |
|---|
Read the documentation.
Here we give a example to render a cuboid using Gaussian ellispoids:
from pytorch3d.renderer import PerspectiveCameras, look_at_view_transform
from VoGE.Converter import Cuboid
from VoGE.Renderer import GaussianRenderer, GaussianRenderSettings, to_white_background
import matplotlib.pyplot as plt
device = 'cuda'
# Create gaussians
gaussians = Cuboid.cuboid_gauss((-1, 1), (-1, 1), (-1, 1), 1000, percentage=0.6, as_obj=True).to(device)
# Create a camera
camera = PerspectiveCameras(focal_length=300, image_size=((256, 256), ), principal_point=((128, 128), ), device=device)
# Create the renderer
render_settings = GaussianRenderSettings(image_size=(256, 256), principal=(128, 128), )
renderer = GaussianRenderer(cameras=camera, render_settings=render_settings)
# Compute camera pose
R, T = look_at_view_transform(dist=6, elev=10, azim=70, device=device)
# Render the Gaussians
frag = renderer(gaussians, R=R, T=T)
# Convert into a image
img = to_white_background(frag, (gaussians.verts + 1) / 3).clamp(0, 1)
plt.imshow(img.squeeze(0).detach().cpu().numpy())
plt.show()
If you find this library is useful, please cite:
@article{wang2022voge,
title={VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for Analysis-by-Synthesis},
author={Wang, Angtian and Wang, Peng and Sun, Jian and Kortylewski, Adam and Yuille, Alan},
journal={arXiv preprint arXiv:2205.15401},
year={2022}
}