[Paper Review] MVSGaussian: Fast Generalizable Gaussian Splatting Reconstruction from Multi-View Stereo

Tianqi Liu, Guangcong Wang+7 more|arXiv (Cornell University)|May 20, 2024

Advanced Vision and Imaging4 citations

TL;DR

MVSGaussian proposes a fast, generalizable 3D Gaussian Splatting framework for novel view synthesis using Multi-View Stereo (MVS) to encode geometry-aware features and decode them into Gaussian parameters. It achieves real-time rendering and sub-60-second per-scene optimization with state-of-the-art view synthesis quality across DTU, LLFF, NeRF Synthetic, and Tanks and Temples datasets.

ABSTRACT

We present MVSGaussian, a new generalizable 3D Gaussian representation approach derived from Multi-View Stereo (MVS) that can efficiently reconstruct unseen scenes. Specifically, 1) we leverage MVS to encode geometry-aware Gaussian representations and decode them into Gaussian parameters. 2) To further enhance performance, we propose a hybrid Gaussian rendering that integrates an efficient volume rendering design for novel view synthesis. 3) To support fast fine-tuning for specific scenes, we introduce a multi-view geometric consistent aggregation strategy to effectively aggregate the point clouds generated by the generalizable model, serving as the initialization for per-scene optimization. Compared with previous generalizable NeRF-based methods, which typically require minutes of fine-tuning and seconds of rendering per image, MVSGaussian achieves real-time rendering with better synthesis quality for each scene. Compared with the vanilla 3D-GS, MVSGaussian achieves better view synthesis with less training computational cost. Extensive experiments on DTU, Real Forward-facing, NeRF Synthetic, and Tanks and Temples datasets validate that MVSGaussian attains state-of-the-art performance with convincing generalizability, real-time rendering speed, and fast per-scene optimization.

Motivation & Objective

• Address the inefficiency of per-scene optimization in 3D Gaussian Splatting, which typically requires minutes for each scene.
• Overcome the challenge of generalizing explicit 3D Gaussian representations to unseen scenes due to high parameter sensitivity.
• Improve generalization beyond implicit NeRF-based methods by incorporating a hybrid rendering strategy that combines splatting and volume rendering.
• Enable fast, high-quality fine-tuning for specific scenes by introducing a multi-view geometric consistent aggregation strategy for initialization.
• Achieve real-time inference and faster optimization compared to existing generalizable NeRF and vanilla 3D-GS methods.

Proposed method

• Leverage Multi-View Stereo (MVS) to estimate depth and encode geometry-aware features for 3D points, using a 2D U-Net to enhance spatial awareness.
• Use an MLP to decode point-wise features into Gaussian parameters (position, scale, rotation, color) in a feed-forward manner.
• Introduce a hybrid Gaussian rendering strategy that combines standard splatting with depth-aware volume rendering to improve generalization.
• Apply a multi-view geometric consistency check to aggregate point clouds from the generalizable model, filtering noisy Gaussians while preserving valid ones.
• Use the aggregated, consistent point cloud as initialization for per-scene optimization, reducing computational cost and accelerating convergence.
• Optimize the model using a differentiable tile-based rasterizer for real-time rendering during inference and fine-tuning.

Experimental results

Research questions

• RQ1 Can a generalizable 3D Gaussian Splatting framework be built using MVS-based geometry reasoning to enable fast inference on unseen scenes?
• RQ2 How does combining splatting and volume rendering improve generalization performance compared to using either method alone?
• RQ3 Can a geometric consistency-based aggregation strategy significantly improve the quality of initialization for per-scene optimization?
• RQ4 What is the trade-off between inference speed, rendering quality, and optimization time when using a generalizable model versus vanilla 3D-GS?
• RQ5 To what extent does the method generalize across diverse scene types, including real-world and synthetic benchmarks?

Key findings

• MVSGaussian achieves state-of-the-art view synthesis performance on DTU, LLFF, NeRF Synthetic, and Tanks and Temples datasets, with PSNR of 28.21, 24.07, 26.46, and 23.28, respectively.
• The method enables real-time rendering with a frame rate of 350 FPS, significantly outperforming NeRF-based methods and matching vanilla 3D-GS speed.
• Per-scene optimization completes in just 45 seconds on the Real Forward-facing dataset, compared to 10 minutes for vanilla 3D-GS and 10 hours for NeRF.
• The hybrid rendering strategy (splatting + volume rendering) improves PSNR by 0.73–1.00 over splatting-only or volume-rendering-only baselines.
• The geometric consistency aggregation strategy achieves 26.98 PSNR on Real Forward-facing, outperforming direct concatenation (26.18) and voxel downsampling (26.72).
• Direct decoding of RGB values instead of spherical harmonics improves generalization, especially on the NeRF Synthetic dataset, where SH decoding reduced PSNR by 1.19 points.