[Paper Review] Periodic Vibration Gaussian: Dynamic Urban Scene Reconstruction and Real-time Rendering

Yu-Rui Chen, Chun Gu+3 more|arXiv (Cornell University)|Nov 30, 2023

Remote Sensing and Land Use4 citations

TL;DR

This paper proposes Periodic Vibration Gaussian (PVG), a unified 3D Gaussian Splatting-based method that models dynamic urban scenes by embedding periodic vibration-based temporal dynamics into static Gaussian points, enabling real-time rendering and superior reconstruction without object-level annotations. PVG achieves 50× faster training and 6000× faster rendering than SUDS, outperforming state-of-the-art methods on Waymo and KITTI benchmarks with significant gains in PSNR, SSIM, and LPIPS.

ABSTRACT

Modeling dynamic, large-scale urban scenes is challenging due to their highly intricate geometric structures and unconstrained dynamics in both space and time. Prior methods often employ high-level architectural priors, separating static and dynamic elements, resulting in suboptimal capture of their synergistic interactions. To address this challenge, we present a unified representation model, called Periodic Vibration Gaussian (PVG). PVG builds upon the efficient 3D Gaussian splatting technique, originally designed for static scene representation, by introducing periodic vibration-based temporal dynamics. This innovation enables PVG to elegantly and uniformly represent the characteristics of various objects and elements in dynamic urban scenes. To enhance temporally coherent and large scene representation learning with sparse training data, we introduce a novel temporal smoothing mechanism and a position-aware adaptive control strategy respectively. Extensive experiments on Waymo Open Dataset and KITTI benchmarks demonstrate that PVG surpasses state-of-the-art alternatives in both reconstruction and novel view synthesis for both dynamic and static scenes. Notably, PVG achieves this without relying on manually labeled object bounding boxes or expensive optical flow estimation. Moreover, PVG exhibits 900-fold acceleration in rendering over the best alternative.

Motivation & Objective

• To address the challenge of reconstructing large-scale, highly dynamic urban scenes with complex spatiotemporal interactions.
• To overcome limitations of prior NeRF-based methods that rely on expensive supervision (e.g., 3D bounding boxes, optical flow) or suffer from slow training and rendering.
• To unify the representation of static and dynamic elements in a single, efficient formulation without explicit object decomposition.
• To improve temporal coherence in reconstruction using sparse training data through novel flow-based smoothing and adaptive control.
• To achieve state-of-the-art performance in novel view synthesis while drastically accelerating training and rendering.

Proposed method

• Extends 3D Gaussian Splatting by introducing time-varying Gaussian positions via periodic vibration: $\widetilde{\bm{\mu}}(t) = \bm{\mu} + \mathbf{A} \sin\left(\frac{2\pi t}{l}\right)$, where $l$ controls cycle length and $\mathbf{A}$ encodes motion amplitude and direction.
• Introduces a scene flow-based temporal smoothing mechanism to enhance motion consistency and reduce flickering in novel view synthesis.
• Employs a position-aware adaptive control strategy to stabilize point behavior in unbounded urban scenes, especially at long distances.
• Incorporates multiple loss terms: depth loss $\mathcal{L}_d$, velocity loss $\mathcal{L}_{\bm{v}}$, opacity loss $\mathcal{L}_o$, and sky refinement for improved geometry and appearance.
• Uses LiDAR supervision and a novel time-dependent opacity decay to enhance reconstruction fidelity and dynamic element isolation.
• Leverages implicit scene flow estimation without requiring ground-truth optical flow or manual annotations.

Experimental results

Research questions

• RQ1 Can a unified 3D Gaussian Splatting formulation effectively represent both static and dynamic elements in large-scale urban scenes without explicit object-level decomposition?
• RQ2 How can temporal coherence be improved in dynamic scene reconstruction when training data is sparse?
• RQ3 Can periodic vibration dynamics outperform linear or stationary motion models in capturing diverse motion patterns in urban scenes?
• RQ4 To what extent can a flow-based temporal smoothing mechanism reduce flickering and improve rendering consistency?
• RQ5 Can the proposed method achieve state-of-the-art novel view synthesis performance while significantly accelerating training and rendering compared to NeRF-based alternatives?

Key findings

• PVG achieves 11.4% higher PSNR, 5.8% higher SSIM, and a 55.1% reduction in LPIPS compared to the SOTA baseline SUDS on the KITTI benchmark.
• On the Waymo Open Dataset, PVG improves PSNR by 9.9% and SSIM by 4.7% over the Mars baseline, with a 20.6% decrease in LPIPS.
• The model achieves a 50× speedup in training and a 6000× speedup in rendering compared to SUDS, the best-performing baseline.
• Abllation studies confirm that the scene flow-based temporal smoothing mechanism improves rendering smoothness and temporal consistency, with PSNR increasing from 27.77 to 28.11 when enabled.
• Optimal cycle length $l$ is critical: too short leads to poor dynamic representation, too long causes ghosting in static regions; the model performs best at intermediate $l$.
• The position-aware adaptive control strategy significantly improves distant view reconstruction, while velocity loss aids in clearer separation of dynamic and static components.