[论文解读] 3D Gaussian Splatting for Real-Time Radiance Field Rendering

Bernhard Kerbl, Georgios Kopanas+2 more|arXiv (Cornell University)|Aug 8, 2023

Advanced Vision and Imaging被引用 26

一句话总结

本文为可微分辐射场引入了3D高斯点云绘制（3D Gaussian splatting），在1080p下实现高质量的新视图合成，具备实时渲染（≥30 fps）和具有竞争力的训练时间。

ABSTRACT

Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (>= 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.

研究动机与目标

• 推动用多张照片或视频捕获的场景实现实时且高质量的新视图合成。
• 使用来自稀疏SfM点的3D高斯来表示场景，以避免对大型MVS数据的依赖。
• 通过自适应密度控制优化3D高斯的属性，以产生紧凑且准确的场景表示。
• 开发一个快速、可微、可见性感知的渲染器，支持各向异性绘制以实现1080p的实时渲染。

提出的方法

• 使用由位置、各向异性协方差和不透明度定义的3D高斯来表示场景。
• 通过可微分投影模型将3D高斯投影到2D扁平圆斑，实现具有Alpha混合的图像形成。
• 以自适应密度控制的交错方式优化高斯参数（位置、通过尺度和旋转的协方差、不透明度、SH颜色系数）。
• 基于视图空间梯度和Alpha阈值在优化过程中对高斯进行密化与裁剪，以在覆盖率和效率之间取得平衡。
• 实现一个快速的基于瓦片的GPU光栅化器，对每张图像预排序扁平斑，支持保持可见性顺序的各向异性绘制，并支持高效的反向传播回传。

Figure 1 . Our method achieves real-time rendering of radiance fields with quality that equals the previous method with the best quality (Barron et al . , 2022 ) , while only requiring optimization times competitive with the fastest previous methods (Fridovich-Keil and Yu et al . , 2022 ; Müller et

实验结果

研究问题

• RQ1 各向异性的3D高斯能否在实现实时渲染的同时，提供与最先进NeRF方法相当的高质量辐射场表示？
• RQ2 自适应密度控制结合快速可微渲染是否使训练时间达到与最快的前代方法竞争同时不牺牲质量？
• RQ3 基于SfM点的初始化（不依赖重型MVS）是否足以为复杂、无界场景获得高质量辐射场？

主要发现

• 在某些训练方案下实现1080p实时渲染，质量等同或优于如Mip-NeRF360等先前的高质量方法。
• 在提供高质量结果的同时，达到与最快的前代方法（如InstantNGP类方法）相媲美的训练时间。
• 用SfM点生成的1–5百万个高斯来表示场景，实现紧凑但准确的场景表示。
• 介绍了一种快速、可微、可见性感知的渲染器，支持各向异性绘制且允许通过每像素的众多绘制进行反向传播。
• 在多个既定数据集（如Tanks&Temples、Deep Blending、Mip-NeRF360）上展示了最先进的视觉质量和实时性能。

Figure 2 . Optimization starts with the sparse SfM point cloud and creates a set of 3D Gaussians. We then optimize and adaptively control the density of this set of Gaussians. During optimization we use our fast tile-based renderer, allowing competitive training times compared to SOTA fast radiance