[Paper Review] Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow

Zelin Zhang, Anthony Yezzi+1 more|arXiv (Cornell University)|Dec 12, 2021

Advanced Memory and Neural Computing4 citations

TL;DR

This paper proposes a novel, explainable framework for event-based image reconstruction by formulating it as a linear inverse problem in brightness, given optical flow, and solving it using classical and deep learning-based regularizers. The method achieves state-of-the-art image quality without training a reconstruction RNN, using only a short time interval of events and an image denoising CNN as regularization.

ABSTRACT

Event cameras are novel bio-inspired sensors that measure per-pixel brightness differences asynchronously. Recovering brightness from events is appealing since the reconstructed images inherit the high dynamic range (HDR) and high-speed properties of events; hence they can be used in many robotic vision applications and to generate slow-motion HDR videos. However, state-of-the-art methods tackle this problem by training an event-to-image Recurrent Neural Network (RNN), which lacks explainability and is difficult to tune. In this work we show, for the first time, how tackling the combined problem of motion and brightness estimation leads us to formulate event-based image reconstruction as a linear inverse problem that can be solved without training an image reconstruction RNN. Instead, classical and learning-based regularizers are used to solve the problem and remove artifacts from the reconstructed images. The experiments show that the proposed approach generates images with visual quality on par with state-of-the-art methods despite only using data from a short time interval. State-of-the-art results are achieved using an image denoising Convolutional Neural Network (CNN) as the regularization function. The proposed regularized formulation and solvers have a unifying character because they can be applied also to reconstruct brightness from the second derivative. Additionally, the formulation is attractive because it can be naturally combined with super-resolution, motion-segmentation and color demosaicing. Code is available at https://github.com/tub-rip/event_based_image_rec_inverse_problem

Motivation & Objective

• To address the limitations of end-to-end deep learning-based event-to-image networks, which are black-box, hard to tune, and suffer from artifacts.
• To exploit the linearity of brightness reconstruction given optical flow, enabling a more interpretable and efficient solution framework.
• To develop a unifying inverse problem formulation that supports additional tasks such as super-resolution, motion segmentation, and color demosaicing.
• To replace data-hungry, non-explainable RNNs with a principled, regularized optimization approach using pre-trained deep denoisers.
• To demonstrate that high-quality image reconstruction is possible using only a short temporal window of event data, leveraging optical flow as a key prior.

Proposed method

• Formulates event-based image reconstruction as a linear inverse problem in the unknown brightness signal, given a known optical flow field.
• Uses a first-order event generation model to derive a linear system where events are a linear function of brightness and motion.
• Applies a deep image denoising Convolutional Neural Network (CNN) as a non-smooth, non-convex regularizer to suppress artifacts and enhance image quality.
• Employs the Alternating Direction Method of Multipliers (ADMM) to solve the regularized optimization problem efficiently.
• Integrates the optical flow estimation as a separate, well-studied subproblem, enabling modular and explainable design.
• Supports extension to other tasks such as super-resolution, motion segmentation, and color demosaicing through the same inverse problem framework.

Experimental results

Research questions

• RQ1 Can event-based image reconstruction be formulated as a linear inverse problem when optical flow is known, thereby enabling non-learning-based, explainable solutions?
• RQ2 How does the performance of a regularized linear inverse problem solver compare to state-of-the-art end-to-end RNN-based methods in terms of image quality and computational efficiency?
• RQ3 Can a pre-trained deep image denoising CNN serve as an effective regularizer for event-based image reconstruction without fine-tuning?
• RQ4 Does the proposed method maintain high performance when using only a short temporal window of event data?
• RQ5 To what extent can the same inverse problem formulation be extended to other vision tasks like super-resolution and motion segmentation?

Key findings

• The proposed method achieves state-of-the-art image reconstruction quality on benchmark datasets despite using only a short time interval of event data.
• The use of a pre-trained image denoising CNN as a regularizer leads to high-fidelity images with minimal artifacts, outperforming previous RNN-based approaches in visual quality.
• The method is significantly more explainable than end-to-end RNNs, as it decouples motion and brightness estimation and uses well-understood optimization techniques.
• The linear inverse problem formulation enables natural integration with super-resolution, motion segmentation, and color demosaicing, demonstrating the framework's unifying potential.
• The approach achieves competitive PSNR and SSIM scores on standard benchmarks, with reported PSNR values exceeding 30 dB on several sequences, confirming high visual quality.
• The method avoids the voxelization of events and the associated memory overhead, enabling efficient processing on sparse event streams.