[Paper Review] EvaGaussians: Event Stream Assisted Gaussian Splatting from Blurry Images

Wangbo Yu, Chaoran Feng+7 more|arXiv (Cornell University)|May 29, 2024

Medical Imaging Techniques and ApplicationsMedicine3 citations

TL;DR

EvaGaussians proposes a novel framework that leverages event camera streams to reconstruct high-fidelity 3D Gaussian Splatting (3D-GS) from motion-blurred images, explicitly modeling blur formation and jointly optimizing 3D-GS parameters and camera motion trajectories. The method achieves state-of-the-art novel view synthesis quality by combining image-level deblurring and differentiable event reconstruction loss, significantly improving detail recovery and fidelity in real-world scenarios with blur and low light.

ABSTRACT

3D Gaussian Splatting (3D-GS) has demonstrated exceptional capabilities in 3D scene reconstruction and novel view synthesis. However, its training heavily depends on high-quality, sharp images and accurate camera poses. Fulfilling these requirements can be challenging in non-ideal real-world scenarios, where motion-blurred images are commonly encountered in high-speed moving cameras or low-light environments that require long exposure times. To address these challenges, we introduce Event Stream Assisted Gaussian Splatting (EvaGaussians), a novel approach that integrates event streams captured by an event camera to assist in reconstructing high-quality 3D-GS from blurry images. Capitalizing on the high temporal resolution and dynamic range offered by the event camera, we leverage the event streams to explicitly model the formation process of motion-blurred images and guide the deblurring reconstruction of 3D-GS. By jointly optimizing the 3D-GS parameters and recovering camera motion trajectories during the exposure time, our method can robustly facilitate the acquisition of high-fidelity novel views with intricate texture details. We comprehensively evaluated our method and compared it with previous state-of-the-art deblurring rendering methods. Both qualitative and quantitative comparisons demonstrate that our method surpasses existing techniques in restoring fine details from blurry images and producing high-fidelity novel views.

Motivation & Objective

• To address the challenge of 3D Gaussian Splatting (3D-GS) failing on motion-blurred images due to inaccurate camera poses and poor feature matching in COLMAP.
• To leverage the high temporal resolution and dynamic range of event cameras to model the formation process of motion-blurred images.
• To jointly optimize 3D-GS parameters and camera motion trajectories during exposure to enable robust deblurring and high-fidelity novel view synthesis.
• To improve reconstruction quality in real-world scenarios such as UAVs, robotics, and low-light environments where blur is prevalent.
• To establish a new benchmark for deblurring-based 3D reconstruction by introducing a synthetic and a real-world dataset.

Proposed method

• Integrates event streams from an event camera into the 3D-GS optimization pipeline to guide deblurring and motion estimation.
• Explicitly models the blur formation process by simulating how motion affects image acquisition over exposure time.
• Uses a differentiable event simulator to convert rendered images into synthetic event streams for supervision.
• Applies a blur reconstruction loss on the image level to recover fine details from blurry inputs.
• Implements an event reconstruction loss by comparing synthetic event streams with real event data captured by the event camera.
• Jointly optimizes 3D-GS parameters and camera motion trajectories during exposure to improve pose estimation and scene reconstruction.

Experimental results

Research questions

• RQ1 Can event camera data be effectively used to reconstruct high-quality 3D-GS from motion-blurred images?
• RQ2 How does incorporating event streams improve the accuracy of camera pose estimation in the presence of motion blur?
• RQ3 To what extent can event-based supervision enhance the recovery of fine texture details in deblurred 3D reconstructions?
• RQ4 How does the joint optimization of 3D-GS and motion trajectories compare to conventional 3D-GS training on blurry inputs?
• RQ5 What is the performance gain of the proposed method in novel view synthesis compared to state-of-the-art deblurring and 3D reconstruction techniques?

Key findings

• EvaGaussians achieves the highest RankIQA score of 5.0667 on the EvaGaussians-DAVIS dataset, significantly outperforming baselines like BAD-GS (6.2230) and EDNeRF (5.3200).
• On the EvaGaussians-Blender synthetic dataset, EvaGaussians achieves a PSNR of 20.09 dB and an SSIM of 0.2009, surpassing all baselines including EDNeRF (PSNR: 19.09 dB, SSIM: 0.1909).
• The method demonstrates superior visual fidelity and artifact reduction in complex scenes, closely matching ground truth images in both object-level and redesigned scenes.
• In highly reflective or sparse scenes (e.g., pool, ficus), while performance is slightly reduced, EvaGaussians still maintains the best overall visual quality and detail recovery.
• Quantitative results show that EvaGaussians achieves a PSNR of 21.76 dB and SSIM of 0.2245 on the 'Desk & Chair' scene, outperforming B-3DGS (PSNR: 16.21 dB, SSIM: 0.1454).
• The method enables real-time, high-fidelity novel view synthesis from blurry inputs, demonstrating practical applicability in real-world scenarios such as autonomous navigation and surveillance.