[论文解读] Text-to-3D using Gaussian Splatting

Zilong Chen, Feng Wang+2 more|arXiv (Cornell University)|Sep 28, 2023

Image Processing and 3D Reconstruction被引用 8

一句话总结

Gsgen 使用三维高斯喷溅（3D Gaussian Splatting）与两阶段优化（几何然后外观），在三维点云扩散先验的引导下，从文本提示生成高质量、三维一致的资产。

ABSTRACT

Automatic text-to-3D generation that combines Score Distillation Sampling (SDS) with the optimization of volume rendering has achieved remarkable progress in synthesizing realistic 3D objects. Yet most existing text-to-3D methods by SDS and volume rendering suffer from inaccurate geometry, e.g., the Janus issue, since it is hard to explicitly integrate 3D priors into implicit 3D representations. Besides, it is usually time-consuming for them to generate elaborate 3D models with rich colors. In response, this paper proposes GSGEN, a novel method that adopts Gaussian Splatting, a recent state-of-the-art representation, to text-to-3D generation. GSGEN aims at generating high-quality 3D objects and addressing existing shortcomings by exploiting the explicit nature of Gaussian Splatting that enables the incorporation of 3D prior. Specifically, our method adopts a progressive optimization strategy, which includes a geometry optimization stage and an appearance refinement stage. In geometry optimization, a coarse representation is established under 3D point cloud diffusion prior along with the ordinary 2D SDS optimization, ensuring a sensible and 3D-consistent rough shape. Subsequently, the obtained Gaussians undergo an iterative appearance refinement to enrich texture details. In this stage, we increase the number of Gaussians by compactness-based densification to enhance continuity and improve fidelity. With these designs, our approach can generate 3D assets with delicate details and accurate geometry. Extensive evaluations demonstrate the effectiveness of our method, especially for capturing high-frequency components. Our code is available at https://github.com/gsgen3d/gsgen

研究动机与目标

• 通过利用可以容纳先验信息的显式三维表示，推动改进的文本到3D生成。
• 通过两阶段优化实现准确几何和高保真外观。
• 提出一个实用的初始化与密度化策略，以提升3D一致性与细节。

提出的方法

• 用一组3D高斯来表示场景并对其进行渐进优化。
• 在几何优化阶段，使用由3D点云扩散先验和2D SDS损失引导来获得粗略、3D一致的形状。
• 在第二阶段通过基于紧凑性准则的迭代密度化来改进外观以丰富细节。
• 用Point-E的3D形状或用户提供的几何对高斯进行初始化，以避免退化。
• 在几何优化阶段对高斯位置应用3D SDS损失以结合3D先验。
• 在细化阶段，依赖2D SDS引导并引入基于紧凑性的密度化，以改善连续性和保真度。

Figure 1: Delicate 3D assets generated using the proposed Gsgen . See our project page gsgen3d.github.io for videos of these images.

实验结果

研究问题

• RQ1 在显式3D先验引导下，3D高斯喷溅是否能成为有效的文本到3D表示？
• RQ2 两阶段优化（带3D先验的几何阶段随后是带密度化的外观细化）是否比仅2D引导在几何和细节上更优？
• RQ3 哪些初始化与密度化策略最能缓解文本到3D生成中的Janus问题和过度平滑？
• RQ4 引入Point-E先验如何影响几何一致性和最终视觉保真度？

主要发现

• 与以往方法相比，Gsgen 生成的3D资产在几何和细节方面更准确。
• 通过Point-E引导引入的3D先验有助于缓解几何塌陷并提升多视角一致性。
• 基于紧凑性的密度化在外观细化阶段提升了几何连续性和细节。
• 以Point-E先验和3D引导进行初始化，相较于随机初始化或仅2D引导，取得更优结果。
• 该方法在捕捉高频成分方面（如纹理和毛发）比若干基线模型更好。

Stable DreamFusion (Tang, 2022 ; Poole et al., 2023 )