[Paper Review] SparseGS: Real-Time 360° Sparse View Synthesis using Gaussian Splatting

Haolin Xiong, Sairisheek Muttukuru+3 more|arXiv (Cornell University)|Nov 30, 2023

Advanced Vision and Imaging4 citations

TL;DR

SparseGS proposes a real-time 360° sparse view synthesis method using 3D Gaussian Splatting that integrates depth priors, diffusion-based generation, and a novel adaptive floater pruning operator to achieve state-of-the-art performance in few-shot settings. It reduces background collapse and floaters, improving LPIPS by up to 30.5% over base 3DGS and 15.6% over NeRF-based methods while maintaining real-time inference on a single GPU.

ABSTRACT

3D Gaussian Splatting (3DGS) has recently enabled real-time rendering of unbounded 3D scenes for novel view synthesis. However, this technique requires dense training views to accurately reconstruct 3D geometry. A limited number of input views will significantly degrade reconstruction quality, resulting in artifacts such as "floaters" and "background collapse" at unseen viewpoints. In this work, we introduce SparseGS, an efficient training pipeline designed to address the limitations of 3DGS in scenarios with sparse training views. SparseGS incorporates depth priors, novel depth rendering techniques, and a pruning heuristic to mitigate floater artifacts, alongside an Unseen Viewpoint Regularization module to alleviate background collapses. Our extensive evaluations on the Mip-NeRF360, LLFF, and DTU datasets demonstrate that SparseGS achieves high-quality reconstruction in both unbounded and forward-facing scenarios, with as few as 12 and 3 input images, respectively, while maintaining fast training and real-time rendering capabilities.

Motivation & Objective

• To address the challenge of coherent 360° novel view synthesis from sparse training views, where existing methods suffer from background collapse and floaters.
• To overcome the limitations of 3D Gaussian Splatting in few-shot settings by introducing explicit, differentiable constraints and a novel pruning mechanism.
• To achieve real-time rendering performance while maintaining high-fidelity results in unbounded, 360° scenes with minimal training data.
• To leverage the explicit nature of 3D Gaussians to enable direct, adaptive manipulation of artifacts like floaters, unlike implicit neural representations.

Proposed method

• The method builds on 3D Gaussian Splatting and introduces a novel adaptive floater pruning operator that identifies and removes high-density, irregularly positioned artifacts in rendered views.
• It incorporates a depth correlation loss using Pearson correlation between predicted and predicted depth maps to regularize depth consistency across views.
• A score distillation sampling (SDS) loss from diffusion models is applied during training to supervise under-constrained regions with low view coverage.
• The training pipeline integrates these components with iterative pruning at 20k and 25k iterations, using learned thresholds (a=97, b=-8) to detect and remove floaters.
• The method uses a patch-based depth correlation loss with 128×128 patches, selecting 50% of patches per iteration to improve generalization.
• Hyperparameters such as λ_depth=0.1 and λ_SDS=5×10⁻⁴ are used to balance the contributions of depth and diffusion supervision.

Figure 1 : Our proposed pipeline incorporates depth and diffusion constraints along with a floater pruning technique to improve few-shot novel view synthesis performance. During training, we render the alpha-blended depth $d^{\text{alpha}}$ and use Pearson correlation to encourage it to align with $

Experimental results

Research questions

• RQ1 Can explicit 3D Gaussian representations be effectively adapted to few-shot 360° novel view synthesis with minimal training views?
• RQ2 Does integrating depth correlation and diffusion-based supervision significantly reduce floaters and background collapse in sparse-view settings?
• RQ3 Can a direct, differentiable pruning operator on 3D Gaussians outperform implicit regularization in neural radiance fields?
• RQ4 How does the combination of explicit constraints and adaptive pruning affect inference speed and rendering quality in unbounded scenes?

Key findings

• SparseGS achieves a 30.5% improvement in LPIPS over base 3D Gaussian Splatting on the Mip-NeRF-360 dataset under few-shot conditions.
• It improves LPIPS by 15.6% compared to NeRF-based methods, demonstrating superior generalization and artifact suppression.
• The ablation study shows that floater pruning contributes the most to PSNR improvement, reducing it by 1.0 point when removed.
• The full model achieves the best depth map quality, combining smoothness from depth correlation loss with accurate background details from pruning and diffusion.
• The method runs at 40 FPS on a single RTX 3090, achieving real-time inference with significantly lower training and inference costs than prior methods.
• Despite strong performance, the method remains sensitive to inaccurate COLMAP point clouds, especially in sparse-view scenarios with fewer than 20 initial points.

Figure 2 : An illustration of how $d^{\text{alpha}}$ and $d^{\text{mode}}$ can disagree. On top, we see an example ray with 4 gaussians along it. On bottom, we see values of $T_{i}$ , $\alpha_{i}$ , and $w_{i}$ along this ray. Note the upward and downward axes both represent positive values and are