[Paper Review] GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation

Quankai Gao, Qiangeng Xu+6 more|arXiv (Cornell University)|Mar 19, 2024

Computer Graphics and Visualization Techniques4 citations

TL;DR

GaussianFlow introduces a novel differentiable framework that links 3D Gaussian dynamics to 2D pixel-level optical flow via splatting, enabling direct supervision of 3D motion through optical flow. This approach significantly improves 4D content generation and novel view synthesis, especially for dynamic scenes, by reducing color drifting and achieving state-of-the-art performance in motion consistency and visual quality.

ABSTRACT

Creating 4D fields of Gaussian Splatting from images or videos is a challenging task due to its under-constrained nature. While the optimization can draw photometric reference from the input videos or be regulated by generative models, directly supervising Gaussian motions remains underexplored. In this paper, we introduce a novel concept, Gaussian flow, which connects the dynamics of 3D Gaussians and pixel velocities between consecutive frames. The Gaussian flow can be efficiently obtained by splatting Gaussian dynamics into the image space. This differentiable process enables direct dynamic supervision from optical flow. Our method significantly benefits 4D dynamic content generation and 4D novel view synthesis with Gaussian Splatting, especially for contents with rich motions that are hard to be handled by existing methods. The common color drifting issue that happens in 4D generation is also resolved with improved Guassian dynamics. Superior visual quality on extensive experiments demonstrates our method's effectiveness. Quantitative and qualitative evaluations show that our method achieves state-of-the-art results on both tasks of 4D generation and 4D novel view synthesis. Project page: https://zerg-overmind.github.io/GaussianFlow.github.io/

Motivation & Objective

• To address the lack of direct dynamic supervision in 4D Gaussian Splatting, especially for monocular or sparse-view inputs.
• To resolve motion ambiguity and color drifting in 4D generation by explicitly supervising 3D Gaussian dynamics using optical flow.
• To improve temporal consistency and realism in 4D content generation and novel view synthesis, particularly in highly dynamic regions.
• To develop a differentiable, efficient method that bridges 3D Gaussian motion with 2D pixel velocities without compromising rendering speed.

Proposed method

• Introduces Gaussian flow as a dense 2D motion representation formed by splatting 3D Gaussian dynamics (translation, rotation, scaling) onto the image plane.
• Uses differentiable splatting to map 3D Gaussian motion into 2D pixel-level flow, enabling end-to-end backpropagation.
• Directly supervises 3D Gaussian dynamics by minimizing the L2 loss between predicted Gaussian flow and ground-truth optical flow.
• Applies flow supervision during both 4D content generation and 4D novel view synthesis to stabilize motion and appearance.
• Combines photometric loss with flow supervision to reduce ambiguity in geometry, appearance, and dynamics under sparse-view conditions.
• Employs a differentiable rendering pipeline that integrates flow supervision into the optimization of 4D Gaussian Splatting fields.

Figure 2 : Between two consecutive frames, a pixel $x_{t_{1}}$ will be pushed towards $x_{t_{1}}\rightarrow x_{i,t_{2}}$ by the 2D Gaussian $i$ ’s motion $i^{t_{1}}\rightarrow i^{t_{2}}$ . We can track $x_{t_{1}}$ in Gaussian $i$ by normalizing it to canonical Gaussian space as $\hat{x}_{i}$ and unn

Experimental results

Research questions

• RQ1 Can 3D Gaussian dynamics be effectively supervised using 2D optical flow in a differentiable and efficient manner?
• RQ2 How does flow-guided supervision improve motion consistency and reduce color drifting in 4D Gaussian Splatting?
• RQ3 Can Gaussian flow enhance both 4D content generation and 4D novel view synthesis, especially in highly dynamic or occluded regions?
• RQ4 How does flow supervision compare to photometric and latent-space supervision (e.g., SDS) in handling motion ambiguity?
• RQ5 Can the method generalize across diverse dynamic scenes with rich motion, including thin structures and fast-moving parts?

Key findings

• The proposed method achieves state-of-the-art results on both 4D content generation and 4D novel view synthesis, with a PSNR of 32.30 on the Dynamic Region Only benchmark.
• Ours achieves 34.22 PSNR on the 4DGS benchmark, outperforming RT-4DGS (32.01) and 4DGS (31.15), demonstrating superior motion realism.
• Without flow supervision, Gaussian motions become inconsistent, leading to non-smooth optical flows and artifacts in novel views, especially on moving parts.
• Flow supervision effectively resolves color drifting in 4D generation by stabilizing dynamic Gaussian behavior across time.
• Visualizations confirm that Gaussian flow aligns with optical flow on both input and novel views only when flow supervision is applied.
• The ablation study shows that flow supervision significantly improves motion quality on thin structures (e.g., bird’s beak) and complex regions (e.g., skull teeth), where photometric loss alone fails.

Figure 3 : Overview of our 4D content generation pipeline. Our model can take an uncalibrated monocular video or video generated from an image as the input. We optimize a 3D Gaussian field by matching the first frame photometrically on reference view and using a 3D-aware SDS loss [ 26 ] to supervise