[Paper Review] SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes

Yihua Huang, Yang-Tian Sun+4 more|arXiv (Cornell University)|Dec 4, 2023

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes SC-GS, a novel method for editable dynamic scene reconstruction using sparse control points and a deformation MLP to model 6 DoF motion fields for 3D Gaussians. By decoupling motion from appearance and enforcing local rigidity via ARAP loss, it achieves high-fidelity novel view synthesis at real-time speeds while enabling intuitive user-controlled motion editing, outperforming prior methods in both quality and efficiency.

ABSTRACT

Novel view synthesis for dynamic scenes is still a challenging problem in computer vision and graphics. Recently, Gaussian splatting has emerged as a robust technique to represent static scenes and enable high-quality and real-time novel view synthesis. Building upon this technique, we propose a new representation that explicitly decomposes the motion and appearance of dynamic scenes into sparse control points and dense Gaussians, respectively. Our key idea is to use sparse control points, significantly fewer in number than the Gaussians, to learn compact 6 DoF transformation bases, which can be locally interpolated through learned interpolation weights to yield the motion field of 3D Gaussians. We employ a deformation MLP to predict time-varying 6 DoF transformations for each control point, which reduces learning complexities, enhances learning abilities, and facilitates obtaining temporal and spatial coherent motion patterns. Then, we jointly learn the 3D Gaussians, the canonical space locations of control points, and the deformation MLP to reconstruct the appearance, geometry, and dynamics of 3D scenes. During learning, the location and number of control points are adaptively adjusted to accommodate varying motion complexities in different regions, and an ARAP loss following the principle of as rigid as possible is developed to enforce spatial continuity and local rigidity of learned motions. Finally, thanks to the explicit sparse motion representation and its decomposition from appearance, our method can enable user-controlled motion editing while retaining high-fidelity appearances. Extensive experiments demonstrate that our approach outperforms existing approaches on novel view synthesis with a high rendering speed and enables novel appearance-preserved motion editing applications. Project page: https://yihua7.github.io/SC-GS-web/

Motivation & Objective

• To address the challenge of high-quality, real-time novel view synthesis in dynamic scenes from monocular videos.
• To enable user-controlled motion editing in 3D scenes while preserving high-fidelity appearance.
• To reduce the complexity and noise of motion modeling in Gaussian splatting by introducing a sparse, learnable motion representation.
• To improve spatial and temporal coherence of motion in dynamic scenes through local rigidity constraints.
• To jointly optimize 3D Gaussians, control point locations, and deformation networks for end-to-end dynamic scene reconstruction.

Proposed method

• Introduces sparse control points (≈512) as a compact basis for 6 DoF motion fields, significantly fewer than the 100K Gaussians.
• Uses a time- and location-conditioned MLP to predict time-varying 6 DoF transformations (quaternion rotation + translation) for each control point.
• Applies local interpolation of control point transformations to drive dense Gaussians, enabling smooth and coherent motion fields.
• Employs an adaptive strategy to dynamically adjust the number and location of control points based on local motion complexity.
• Introduces an ARAP (As-Rigid-As-Possible) loss to enforce local rigidity and spatial continuity in the motion of control points.
• Jointly optimizes 3D Gaussian parameters, canonical control point locations, and the deformation MLP using monocular video input.

Experimental results

Research questions

• RQ1 Can a sparse control point representation effectively model complex, dynamic scene motions while maintaining high rendering quality?
• RQ2 How does a deformation MLP conditioned on time and location improve motion generalization and coherence compared to direct Gaussian motion prediction?
• RQ3 To what extent does enforcing local rigidity via ARAP loss reduce motion artifacts and improve temporal consistency?
• RQ4 Can the explicit motion representation via control points enable intuitive, user-guided motion editing beyond the training sequence?
• RQ5 How does the method compare quantitatively and qualitatively to SOTA approaches in novel view synthesis and motion editing?

Key findings

• On the D-NeRF dataset, SC-GS achieves a PSNR of 43.307, SSIM of 0.9976, and LPIPS of 0.0063, outperforming baselines with and without control points.
• Ablation studies show that removing control points reduces PSNR by 4.795 points, demonstrating their critical role in generalization and noise reduction.
• Removing the ARAP loss increases LPIPS from 0.0063 to 0.0067 and introduces motion artifacts, confirming its effectiveness in enforcing rigidity.
• The method generalizes well to unseen motion patterns, as shown in motion editing results where edited trajectories remain smooth and plausible.
• SC-GS maintains high rendering speed while achieving state-of-the-art visual quality, even on challenging scenes with specular reflections.
• The approach enables user-editable motion editing by manipulating control points, a capability absent in prior NeRF- and Gaussian-based methods.