[论文解读] Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting

Zeyu Yang, Hongye Yang+2 more|arXiv (Cornell University)|Oct 16, 2023

Advanced Vision and Imaging被引用 33

一句话总结

论文提出了一个4D高斯投射(4DGS)表示，用于动态图像的端到端可训练、实时和高分辨率动态图视图的光真实渲染，通过对时空4D高斯和随时间演化的外观建模，结合4D球圆柱谐波。

ABSTRACT

Reconstructing dynamic 3D scenes from 2D images and generating diverse views over time is challenging due to scene complexity and temporal dynamics. Despite advancements in neural implicit models, limitations persist: (i) Inadequate Scene Structure: Existing methods struggle to reveal the spatial and temporal structure of dynamic scenes from directly learning the complex 6D plenoptic function. (ii) Scaling Deformation Modeling: Explicitly modeling scene element deformation becomes impractical for complex dynamics. To address these issues, we consider the spacetime as an entirety and propose to approximate the underlying spatio-temporal 4D volume of a dynamic scene by optimizing a collection of 4D primitives, with explicit geometry and appearance modeling. Learning to optimize the 4D primitives enables us to synthesize novel views at any desired time with our tailored rendering routine. Our model is conceptually simple, consisting of a 4D Gaussian parameterized by anisotropic ellipses that can rotate arbitrarily in space and time, as well as view-dependent and time-evolved appearance represented by the coefficient of 4D spherindrical harmonics. This approach offers simplicity, flexibility for variable-length video and end-to-end training, and efficient real-time rendering, making it suitable for capturing complex dynamic scene motions. Experiments across various benchmarks, including monocular and multi-view scenarios, demonstrate our 4DGS model's superior visual quality and efficiency.

研究动机与目标

• 通过在统一的4D体积中捕捉时空结构，动实时、光真实的动态图渲染从2D图像出发的动机。
• 提出具有明确几何和外观的4D高斯原语用于动态图景，使端到端训练和实时渲染成为可能。
• 通过4D球圆柱谐波引入随时间演化的外观模型，以捕捉随时间变化的基于视角的颜色变化。
• 在与先前方法的对比中，在真实与合成的多样动态图数据集上展示更优的视觉质量和效率。

提出的方法

• 将动态图景表示为一组具有均值、时空协方差和随时间演化颜色的4D高斯。
• 将4D协方差参数化为 Sigma = R S S^T R^T，其中4D旋转R（通过左右各向同性旋转）和对空 间与时间的对角缩放S。
• 推导条件高斯 p(xyz|t) 和边际 p(t)，在投影到图像平面时对每个高斯渲染一个二维splat，并在时间上进行积分。
• 将(x,y,z)与t作为一个连贯的4D高斯的分量来联合建模时空，使端到端优化与 radiance-splat 渲染成为可能。
• 用4D球圆柱谐波(4DSH)表示基于视角的颜色，以捕捉随时间演化的外观依赖于视点和时间的变化。
• 通过端到端的渲染损失进行训练，使用随时间采样的批次以降低时间颤动，并在时空中进行密化与密度控制。

Figure 1: Schematic illustration of the proposed 4DGS. This diagram illustrates why our 4D primitive is naturally suitable for representing dynamic scenes. Within our framework, the transition from 4D Gaussian to 2D Planar Gaussian can conceptually correspond to the process where a dynamic scene tra

实验结果

研究问题

• RQ1 一个统一的4D高斯原语能否捕捉动态图景的时空结构，从而实现光真实、实时的视图合成？
• RQ2 4D旋转与4D SH为基础的外观模型是否比基于3D或时间分离的表示在动态图景渲染质量上有所提升？
• RQ3 是否能够通过4D高斯光栅化管线实现对单视角和多视点动态图数据集的端到端训练与实时渲染？

主要发现

• 4D高斯原语与4D旋转可以有效建模动态图景，并实现实时高保真的渲染。
• 4D球圆柱谐波(4DSH)捕捉随时间演化且随视角变化的外观，提升视觉质量。
• 在Plenoptic Video与D-NeRF数据集上，4DGS在实时帧率下实现的PSNR、SSIM/DPSSIM类指标与LPIPS优于先前方法。
• 消融研究表明，联合建模时空的4D旋转与时间耦合外观优于时间独立的扩展。
• 该方法支持对整段视频进行端到端训练，而非逐帧优化，从而实现可扩展的动态图景合成。

Figure 2: Rendering pipeline of our 4DGS. Given a time $t$ and view $\mathcal{I}$ , each 4D Gaussian is first decomposed into a conditional 3D Gaussian and a marginal 1D Gaussian. Subsequently, the conditional 3D Gaussian is projected to a 2D splat. Finally, we integrate the planar conditional Gauss