[Paper Review] DeblurGS: Gaussian Splatting for Camera Motion Blur

Jeongtaek Oh, Jaeyoung Chung+2 more|arXiv (Cornell University)|Apr 17, 2024

Advanced Image Processing Techniques4 citations

TL;DR

DeblurGS is the first method to optimize sharp 3D Gaussian Splatting from motion-blurred images by jointly estimating camera motion and refining scene geometry. It achieves state-of-the-art deblurring and novel view synthesis performance even with noisy camera poses from Structure-from-Motion, using a novel Gaussian Densification Annealing strategy and sub-frame alignment for robust optimization on real-world smartphone videos.

ABSTRACT

Although significant progress has been made in reconstructing sharp 3D scenes from motion-blurred images, a transition to real-world applications remains challenging. The primary obstacle stems from the severe blur which leads to inaccuracies in the acquisition of initial camera poses through Structure-from-Motion, a critical aspect often overlooked by previous approaches. To address this challenge, we propose DeblurGS, a method to optimize sharp 3D Gaussian Splatting from motion-blurred images, even with the noisy camera pose initialization. We restore a fine-grained sharp scene by leveraging the remarkable reconstruction capability of 3D Gaussian Splatting. Our approach estimates the 6-Degree-of-Freedom camera motion for each blurry observation and synthesizes corresponding blurry renderings for the optimization process. Furthermore, we propose Gaussian Densification Annealing strategy to prevent the generation of inaccurate Gaussians at erroneous locations during the early training stages when camera motion is still imprecise. Comprehensive experiments demonstrate that our DeblurGS achieves state-of-the-art performance in deblurring and novel view synthesis for real-world and synthetic benchmark datasets, as well as field-captured blurry smartphone videos.

Motivation & Objective

• To address the challenge of reconstructing sharp 3D scenes from motion-blurred images when camera pose initialization from Structure-from-Motion (SfM) is severely inaccurate due to blur.
• To overcome the limitations of existing NeRF-based deblurring methods that rely on known or clean camera poses, which are unrealistic in real-world settings.
• To enable practical 3D reconstruction from real-world, fast-moving, blurry smartphone videos where SfM fails to produce reliable initial poses.
• To develop a robust optimization framework that jointly refines camera motion and 3D scene geometry using differentiable rendering and blur simulation.
• To demonstrate that 3D Gaussian Splatting can be effectively leveraged for deblurring, even under poor initial conditions, by introducing new training strategies.

Proposed method

• DeblurGS jointly optimizes 3D Gaussian Splatting and 6-DoF camera motion for each blurry input image by simulating blur through accumulated renderings along estimated motion trajectories.
• It synthesizes blurry renderings by sampling sub-frames along the continuous camera motion trajectory using learnable alignment parameters, enabling precise blur modeling.
• The method employs a differentiable rendering pipeline that minimizes the L2 loss between synthesized blurry images and real input blurry observations.
• A novel Gaussian Densification Annealing strategy gradually increases the density threshold for Gaussian creation, preventing spurious Gaussians from forming during early training when camera motion is inaccurate.
• Temporal smoothness loss is applied between adjacent sub-frame renderings to suppress jittering artifacts and ensure stable optimization.
• Gamma correction is applied during blur synthesis to better simulate real camera response, improving rendering fidelity and optimization convergence.

Experimental results

Research questions

• RQ1 Can 3D Gaussian Splatting be effectively adapted for deblurring motion-blurred images when camera poses are initialized with high error from SfM?
• RQ2 How can the optimization of 3D Gaussian Splatting be stabilized during early training stages when camera motion is inaccurate and noisy?
• RQ3 What role does sub-frame sampling along the motion trajectory play in improving blur synthesis accuracy and final deblurring quality?
• RQ4 How does the Gaussian Densification Annealing strategy affect the suppression of floating artifacts caused by erroneous initial poses?
• RQ5 Can DeblurGS achieve state-of-the-art performance on real-world, fast-moving smartphone videos where blur and pose errors are severe?

Key findings

• DeblurGS achieves state-of-the-art performance in both deblurring and novel view synthesis across synthetic and real-world datasets, outperforming all prior NeRF-based and 3DGS-based methods.
• On the ExBlur-NP benchmark, DeblurGS achieves a PSNR of 32.15, SSIM of 0.942, and LPIPS of 0.052, matching the performance of ExBluRF with clean poses while operating from noisy SfM initialization.
• The method maintains high-quality reconstruction even with COLMAP-initialized poses from real smartphone videos, successfully recovering sharp details in complex scenes.
• Abllation studies confirm that all components—temporal smoothness loss, Gaussian Densification Annealing, alignment parameters, and gamma correction—are essential for optimal performance.
• In real-world video experiments, DeblurGS successfully reconstructs sharp 3D scenes from 30 fps smartphone videos captured in under 6 seconds, demonstrating practical applicability.
• The Gaussian Densification Annealing strategy significantly reduces floating artifacts and improves reconstruction quality by preventing Gaussians from forming at incorrect locations during early optimization.