With the widespread usage of VR devices and contents, demands for 3D scene generation techniques become more popular. Existing 3D scene generation models, however, limit the target scene to specific domain, primarily due to their training strategies using 3D scan dataset that is far from the real-world. To address such limitation, we propose LucidDreamer, a domain-free scene generation pipeline by fully leveraging the power of existing large-scale diffusion-based generative model. Our LucidDreamer has two alternate steps: Dreaming and Alignment. First, to generate multi-view consistent images from inputs, we set the point cloud as a geometrical guideline for each image generation. Specifically, we project a portion of point cloud to the desired view and provide the projection as a guidance for inpainting using the generative model. The inpainted images are lifted to 3D space with estimated depth maps, composing a new points. Second, to aggregate the new points into the 3D scene, we propose an aligning algorithm which harmoniously integrates the portions of newly generated 3D scenes. The finally obtained 3D scene serves as initial points for optimizing Gaussian splats. LucidDreamer produces Gaussian splats that are highly-detailed compared to the previous 3D scene generation methods, with no constraint on domain of the target scene.
随着VR设备和内容的广泛使用，对3D场景生成技术的需求变得越来越流行。然而，现有的3D场景生成模型限制了目标场景到特定领域，主要是因为它们使用远离现实世界的3D扫描数据集进行训练。为了解决这一限制，我们提出了LucidDreamer，这是一种域自由的场景生成管道，充分利用现有的大规模基于扩散的生成模型的能力。我们的LucidDreamer有两个交替步骤：梦境和对齐。首先，为了从输入生成多视图一致的图像，我们将点云设置为每个图像生成的几何指南。具体来说，我们将点云的一部分投影到期望的视图，并将投影作为使用生成模型进行修补的指南。用估计的深度图提升到3D空间的修补图像，组成了新的点。其次，为了将新点聚合到3D场景中，我们提出了一种对齐算法，和谐地集成了新生成的3D场景部分。最终获得的3D场景作为优化高斯溅射的初始点。与以前的3D场景生成方法相比，LucidDreamer产生的高斯溅射细节丰富，且目标场景的领域没有限制。