Panoptic 3D Scene Reconstruction from a Single RGB Image
Manuel Dahnert, Ji Hou, Matthias Nießner, Angela Dai
Neural Information Processing Systems (NeurIPS) - 2021

If you find this work useful for your research, please consider citing

@inproceedings{dahnert2021panoptic,
  title={Panoptic 3D Scene Reconstruction From a Single RGB Image},
  author={Dahnert, Manuel and Hou, Ji and Nie{\ss}ner, Matthias and Dai, Angela},
  booktitle={Thirty-Fifth Conference on Neural Information Processing Systems},
  year={2021}
}

Understanding 3D scenes from a single image is fundamental to a wide variety of tasks, such as for robotics, motion planning, or augmented reality. Existing works in 3D perception from a single RGB image tend to focus on geometric reconstruction only, or geometric reconstruction with semantic segmentation or instance segmentation. Inspired by 2D panoptic segmentation, we propose to unify the tasks of geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation into the task of panoptic 3D scene reconstruction - from a single RGB image, predicting the complete geometric reconstruction of the scene in the camera frustum of the image, along with semantic and instance segmentations. We thus propose a new approach for holistic 3D scene understanding from a single RGB image which learns to lift and propagate 2D features from an input image to a 3D volumetric scene representation. We demonstrate that this holistic view of joint scene reconstruction, semantic, and instance segmentation is beneficial over treating the tasks independently, thus outperforming alternative approaches.

The code was tested with the following configuration:

• Ubuntu 20.04
• Python 3.8
• Pytorch 1.7.1
• CUDA 10.2
• Minkowski Engine 0.5.1, fork
• Mask RCNN Benchmark
• Nvidia 2080 Ti, 11GB

# Basic conda enviromnent: Creates new conda environment `panoptic`
conda env create --file environment.yaml
conda activate panoptic

Follow the official instructions to install the maskrcnn-benchmark repo.

Follow the instructions to compile our forked Minkowski Engine version from source.

Finally, compile this library.

# Install library
cd lib/csrc/
python setup.py install

To run the method on a 3D-Front sample run python tools/test_net_single_image.py with the pre-trained checkpoint (see table below).

python tools/test_nest_single_image.py -i <path_to_input_image> -o <output_path>

The 3D-FRONT indoor datasets consists of 6,813 furnished apartments.
We use Blender-Proc [2] to render photo-realistic images from individual rooms. We use version from 2020-06-14 of the data.

We provide the preprocessed 3D-Front data, please see the following table for links to the main zips.
Extract the downloaded data into data/front3d/ or adjust the root data path lib/config/paths_catalog.py.
By downloading our derived work from the original 3D-Front you accept their original Terms of Use.

• We replace all walls and ceilings and "re-draw" them in order to close holes in the walls, e.g. empty door frames or windows.
  For the ceiling we use the same geometry as the floor plane to have a closed room envelope.
• We remove following mesh categories: "WallOuter", "WallBottom", "WallTop", "Pocket", "SlabSide", "SlabBottom", "SlabTop", "Front", "Back", "Baseboard", "Door", "Window", "BayWindow", "Hole", "WallInner", "Beam"
• During rendering, we only render geometry which is assigned to the current room
• We sample each individual (non-empty) room
  • num max tries: 50,000
  • num max samples per room: 50
• Camera:
  • we fix the camera height at 0.75m and choose a forward-looking camera angle (similar to the original frames in 3D-Front)

<scene_id>/            
    ├── rgb_<frame_id>.png                  # Color image: 320x240x3
    ├── depth_<frame_id>.exr                # Depth image: 320x240x1
    ├── segmap_<frame_id>.mapped.npz        # 2D Segmentation: 320x240x2, with 0: pre-mapped semantics, 1: instances
    ├── geometry_<frame_id>.npz             # 3D Geometry: 256x256x256x1, truncated, (unsigned) distance field at 3cm voxel resolution and 12 voxel truncation.
    ├── segmentation_<frame_id>.mapped.npz  # 3D Segmentation: 256x256x256x2, with 0: pre-mapped semantics & instances
    ├── weighting_<frame_id>.mapped.npz     # 3D Weighting mask: 256x256x256x1
    ...

In total, we generate 197,352 frames of which 134,389 were used for this project. We filter out frames, which have an inconsistent number of 2D and 3D instances.

For the 3D generation we use a custom C++ pipeline which loads the sampled camera poses, room layout mesh, and the scene objects. The geometry is cropped to the camera frustum, such that only geometry within the frustum contributes to the DF calculation. It generates 3D unsigned distance fields at 3cm resolution together with 3D semantic and instance segmentation.

• 2022-04-28:
  • Move data to new location with better connectivity. Please use the links in the table above to download the data.
  • Provide additional 3D-Front data samples, which were generated but not used for this project.
  • Rename 3D-Front checkpoint from panoptic_front3d_v2.pth to panoptic_front3d.pth. Note: This checkpoint evaluates to ~43% PRQ, compared to the stated 46.77% PRQ reported in the paper - we are currently investigating this gap.
  • Some 3D segmentation samples did break during conversion to the provided npz format due to an integer overflow of the labels. - Sorry for any inconveniences caused.
  • Add evaluation code.
  • Fix a voxel shift in the backprojection layer.
• 2022-04-05: Add initial Matterport release (dataloader, file lists, example sample).
• 2022-03-03: Add script to evaluate a single image.
• 2022-02-12: Bug fixes
• 2021-12-22: Initial commit of cleaned up code.

1. Fu et al. - 3d-Front: 3d Furnished Rooms with Layouts and Semantics
2. Denninger et al. - BlenderProc