Skip to main content
QUICK REVIEW

[论文解读] LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes

Jaeyoung Chung, Suyong Lee|arXiv (Cornell University)|Nov 22, 2023
Generative Adversarial Networks and Image SynthesisComputer Science被引用 12
一句话总结

LucidDreamer 通过 Dreaming 和 Alignment 构建多视图一致的点云,从文本、RGB 或 RGBD 输入生成多样且高质量的 3D 场景,然后优化高斯光斑以进行渲染。它利用 Stable Diffusion 和深度估计实现领域无关的 3D 场景创建。

ABSTRACT

With the widespread usage of VR devices and contents, demands for 3D scene generation techniques become more popular. Existing 3D scene generation models, however, limit the target scene to specific domain, primarily due to their training strategies using 3D scan dataset that is far from the real-world. To address such limitation, we propose LucidDreamer, a domain-free scene generation pipeline by fully leveraging the power of existing large-scale diffusion-based generative model. Our LucidDreamer has two alternate steps: Dreaming and Alignment. First, to generate multi-view consistent images from inputs, we set the point cloud as a geometrical guideline for each image generation. Specifically, we project a portion of point cloud to the desired view and provide the projection as a guidance for inpainting using the generative model. The inpainted images are lifted to 3D space with estimated depth maps, composing a new points. Second, to aggregate the new points into the 3D scene, we propose an aligning algorithm which harmoniously integrates the portions of newly generated 3D scenes. The finally obtained 3D scene serves as initial points for optimizing Gaussian splats. LucidDreamer produces Gaussian splats that are highly-detailed compared to the previous 3D scene generation methods, with no constraint on domain of the target scene. Project page: https://luciddreamer-cvlab.github.io/

研究动机与目标

  • 推动无领域限制的 3D 场景生成,超越 3D 扫描和领域受限的数据。
  • 利用大规模扩散模型生成多视图一致的图像,为 3D 构建提供引导。
  • 开发一个两阶段流水线(Dreaming 与 Alignment)以构建统一的 3D 点云。
  • 从构建的点云中优化高斯点绘表示以实现高质量渲染。

提出的方法

  • Dreaming:沿轨迹移动相机、投影可见的点云区域、用 Stable Diffusion 修复被掩模的区域、估计深度,并提升到 3D 以扩展点云。
  • Alignment:通过沿着相机光线移动点并插值深度,将新点与现有云对齐,从而创建一个光滑、连贯的场景。
  • Construct a complete point cloud P_N by repeating Dreaming and Alignment to enable multi-view consistency.
  • Render and refine the scene by training a 3D Gaussian splatting model using the constructed point cloud and projected images, with additional synthetic views to improve coverage.
  • Depths are adjusted via scale factors d_i to minimize discrepancy between new and existing 3D points, ensuring coherent 3D geometry.

实验结果

研究问题

  • RQ1一个无领域限制的流水线是否能够从多种输入类型(文本、RGB、RGBD)可靠地生成多视图一致且高质量的 3D 场景?
  • RQ2如何将基于扩散的修复、深度估计与对齐整合起来,以生成一个连贯且高保真的 3D Gaussian splatting 场景?
  • RQ3与仅点云表示相比,Gaussian splatting 的优化是否提升了渲染质量并填补深度空洞?

主要发现

  • LucidDreamer 在多样领域(现实、动漫、乐高)上产生的 3D 场景质量高于先前受域限制的方法。
  • Dreaming 与 Alignment 过程实现多视图一致性,并将新生成的场景部分无缝整合到一个单一的 3D 模型中。
  • Gaussian splatting 的最终化填补了空洞并提升了渲染真实感,超出初始点云。
  • 基于 CLIP 的指标进行的定量对比在多个数据集上显示相对于 RGBD2 的感知质量和一致性更有利。
  • 该方法支持多种输入类型(文本、RGB、RGBD),并在生成过程中允许混合输入条件。

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。