[Paper Review] Robust Gaussian Splatting

François Darmon, Lorenzo Porzi+2 more|arXiv (Cornell University)|Apr 5, 2024

Advanced Statistical Process MonitoringDecision Sciences3 citations

TL;DR

This paper proposes a robust 3D Gaussian Splatting framework that jointly models motion blur as a Gaussian distribution over camera poses, enabling unified refinement of camera poses and blur correction. It further introduces defocus blur compensation via 2D covariance offsets and per-image color transformation to handle lighting and white balance variations, achieving state-of-the-art performance on real-world (ScanNet++) and synthetic (Deblur-NeRF) benchmarks with significant PSNR and SSIM gains.

ABSTRACT

In this paper, we address common error sources for 3D Gaussian Splatting (3DGS) including blur, imperfect camera poses, and color inconsistencies, with the goal of improving its robustness for practical applications like reconstructions from handheld phone captures. Our main contribution involves modeling motion blur as a Gaussian distribution over camera poses, allowing us to address both camera pose refinement and motion blur correction in a unified way. Additionally, we propose mechanisms for defocus blur compensation and for addressing color in-consistencies caused by ambient light, shadows, or due to camera-related factors like varying white balancing settings. Our proposed solutions integrate in a seamless way with the 3DGS formulation while maintaining its benefits in terms of training efficiency and rendering speed. We experimentally validate our contributions on relevant benchmark datasets including Scannet++ and Deblur-NeRF, obtaining state-of-the-art results and thus consistent improvements over relevant baselines.

Motivation & Objective

• To improve the robustness of 3D Gaussian Splatting (3DGS) for real-world applications involving low-quality, handheld captures.
• To address common error sources such as motion blur, defocus blur, imperfect camera poses, and color inconsistencies due to lighting or camera settings.
• To maintain the training efficiency and rendering speed of 3DGS while enhancing fidelity under noisy, distorted input data.
• To validate the method on real-world datasets like ScanNet++ and synthetic benchmarks like Deblur-NeRF under realistic evaluation protocols.

Proposed method

• Models motion blur as a Gaussian distribution over camera poses, enabling joint optimization of pose and blur correction during training.
• Introduces a 2D covariance offset mechanism to the Gaussians during projection to compensate for defocus blur in the image plane.
• Incorporates a learnable per-image affine color transformation to correct for ambient light, shadows, and white balance variations.
• Absorbs the color transformation into spherical harmonics at inference, enabling compatibility with standard 3DGS renderers.
• Uses test-time adaptation to align predicted poses and colors with ground truth for evaluation, avoiding overfitting.
• Employs a blurriness measure (decreasing maximum gradient magnitude) to select high-quality test views for evaluation on ScanNet++.

Figure 1 : Hand-held phone captures (top left, from ScanNet++) can be challenging to reconstruct with 3D Gaussian Splatting (top center), due to inter-frame color inconsistencies (bottom left), motion blur and defocus blur (bottom right). We show how these factors can explicitly and easily be modele

Experimental results

Research questions

• RQ1 Can motion blur and camera pose errors be jointly corrected within the 3DGS framework using a probabilistic formulation?
• RQ2 How effective is modeling defocus blur via 2D covariance offsets in improving reconstruction quality?
• RQ3 To what extent can per-image color transformations mitigate color inconsistencies from ambient lighting and camera settings?
• RQ4 Does the proposed method achieve state-of-the-art performance on real-world and synthetic benchmarks under realistic evaluation conditions?

Key findings

• The full method achieves a PSNR of 24.08 and SSIM of 0.850 on the ScanNet++ benchmark, outperforming baseline 3DGS by over 2.35 PSNR points.
• Color transformation and pose optimization together yield a 2.37 PSNR gain over baseline, with color correction alone causing a 1.37 PSNR drop if omitted.
• Motion blur modeling alone achieves 24.08 PSNR, while adding defocus blur modeling yields no further improvement, likely due to low defocus blur prevalence in ScanNet++.
• On the Deblur-NeRF synthetic dataset, the method achieves 29.24 PSNR and 0.9168 SSIM for defocus blur, outperforming standard 3DGS (24.46 PSNR) and Deblur-GS (29.43 PSNR).
• The method significantly improves performance on real-world data, with PSNR gains of 2.12–2.37 points over baseline 3DGS across ablation settings.
• Defocus blur modeling proves beneficial on Deblur-NeRF but less so on ScanNet++, where motion blur dominates and defocus blur is rare.

Figure 2 : In 3DGS, defocus blur (left), color inconsistencies (center) and motion blur (right) can be modeled as simple transformations applied to the 3D Gaussian primitives (Sec. 4 ). This allows us to estimate per-camera motion, appearance and focus parameters, which can be factored out to recove