[Paper Review] InstantSplat: Sparse-view Gaussian Splatting in Seconds

Zhiwen Fan, Wenyan Cong+10 more|arXiv (Cornell University)|Mar 29, 2024

Spectroscopy Techniques in Biomedical and Chemical Research9 citations

TL;DR

InstantSplat unifies pose-free, sparse-view 3D Gaussian Splatting with a dense stereo prior to reconstruct and render large-scale scenes in under a minute, without precomputed camera intrinsics/extrinsics.

ABSTRACT

While neural 3D reconstruction has advanced substantially, its performance significantly degrades with sparse-view data, which limits its broader applicability, since SfM is often unreliable in sparse-view scenarios where feature matches are scarce. In this paper, we introduce InstantSplat, a novel approach for addressing sparse-view 3D scene reconstruction at lightning-fast speed. InstantSplat employs a self-supervised framework that optimizes 3D scene representation and camera poses by unprojecting 2D pixels into 3D space and aligning them using differentiable neural rendering. The optimization process is initialized with a large-scale trained geometric foundation model, which provides dense priors that yield initial points through model inference, after which we further optimize all scene parameters using photometric errors. To mitigate redundancy introduced by the prior model, we propose a co-visibility-based geometry initialization, and a Gaussian-based bundle adjustment is employed to rapidly adapt both the scene representation and camera parameters without relying on a complex adaptive density control process. Overall, InstantSplat is compatible with multiple point-based representations for view synthesis and surface reconstruction. It achieves an acceleration of over 30x in reconstruction and improves visual quality (SSIM) from 0.3755 to 0.7624 compared to traditional SfM with 3D-GS.

Motivation & Objective

• Motivate robust novel view synthesis (NVS) in unconstrained, sparse-view settings without relying on precomputed camera intrinsics/extrinsics.
• Combine explicit 3D Gaussian Splatting with end-to-end dense stereo priors to enable fast, pose-free scene reconstruction.
• Provide a streamlined optimization pipeline that initializes geometry and jointly optimizes Gaussian attributes and camera parameters.
• Demonstrate scalability to large outdoor scenes with improved rendering quality and pose accuracy.

Proposed method

• Propose InstantSplat, a two-stage pipeline: Coarse Geometric Initialization (CGI) to obtain globally aligned point maps and initial poses from a pre-trained dense stereo model (DUSt3R).
• Utilize DUSt3R to generate per-view point maps and confidence maps from uncalibrated/unposed images, and recover camera intrinsics via Weiszfeld-based focal estimation.
• Perform Fast 3D-Gaussian Optimization (F-3DGO) that jointly refine 3D Gaussian attributes and camera parameters starting from a globally aligned 3D prior, with pose regularization.
• Render via 3D Gaussian Splatting (3D-GS) with SH-based color encoding and view-direction dependent coefficients.
• Align test-view poses by freezing the trained 3D-GS model and optimizing test-view camera poses to minimize photometric discrepancy, following NeRFmm-inspired strategy.

Experimental results

Research questions

• RQ1 How can NVS be performed in pose-free, sparse-view scenarios without relying on COLMAP-type precomputations?
• RQ2 Can end-to-end dense stereo priors be effectively integrated with Gaussian splatting to produce high-quality renderings with accurate pose estimations in large-scale outdoor scenes?
• RQ3 What is the impact of initializing with globally aligned point maps on the optimization efficiency and final rendering quality?
• RQ4 How do joint optimizations of Gaussian attributes and camera parameters compare to conventional pose-dependent methods in terms of speed and accuracy?

Key findings

• InstantSplat achieves rendering quality improvements (e.g., SSIM up to 0.89 in Tanks & Temples and MVImgNet) while significantly reducing optimization time (under 1 minute).
• Pose estimation accuracy improves substantially, with ATE reductions (e.g., from 0.055 to 0.011 in Tanks & Temples) and RPE improvements across evaluated scenes.
• The method outperforms pose-free baselines such as CF-3DGS and Nope-NeRF on both rendering quality (PSNR/SSIM/LPIPS) and pose metrics, while avoiding artifacts from unstable pose estimates.
• Using globally aligned point maps as priors reduces reliance on complex adaptive density controls in 3DGS and speeds up optimization.
• Joint optimization of Gaussian attributes with camera parameters, under pose regularization, yields sharper renderings and more robust alignment for test views.