[Paper Review] HDR Denoising and Deblurring by Learning Spatio-temporal Distortion Models

Uğur Çoğalan, Mojtaba Bemana+3 more|arXiv (Cornell University)|Dec 22, 2020

Advanced Image Processing Techniques108 references4 citations

TL;DR

This paper proposes a deep learning method for reconstructing high-dynamic range (HDR) video from dual-exposure sensors by learning a Clean → Distorted distortion model instead of the conventional Distorted → Clean approach. By synthesizing realistic noise, motion blur, and row/column artifacts from limited clean HDR data and leveraging high-speed LDR video for motion blur simulation, the method achieves state-of-the-art HDR denoising and deblurring, significantly improving both standalone performance and the accuracy of downstream methods like BM3D when retrained on its synthetic data.

ABSTRACT

We seek to reconstruct sharp and noise-free high-dynamic range (HDR) video from a dual-exposure sensor that records different low-dynamic range (LDR) information in different pixel columns: Odd columns provide low-exposure, sharp, but noisy information; even columns complement this with less noisy, high-exposure, but motion-blurred data. Previous LDR work learns to deblur and denoise (DISTORTED->CLEAN) supervised by pairs of CLEAN and DISTORTED images. Regrettably, capturing DISTORTED sensor readings is time-consuming; as well, there is a lack of CLEAN HDR videos. We suggest a method to overcome those two limitations. First, we learn a different function instead: CLEAN->DISTORTED, which generates samples containing correlated pixel noise, and row and column noise, as well as motion blur from a low number of CLEAN sensor readings. Second, as there is not enough CLEAN HDR video available, we devise a method to learn from LDR video in-stead. Our approach compares favorably to several strong baselines, and can boost existing methods when they are re-trained on our data. Combined with spatial and temporal super-resolution, it enables applications such as re-lighting with low noise or blur.

Motivation & Objective

• To address the lack of clean HDR video and time-consuming capture of distorted sensor readings in HDR reconstruction.
• To overcome the limitations of existing methods that assume simple noise or blur models, which do not apply to dual-exposure sensors with correlated noise and alternating blur.
• To develop a data-efficient training strategy using a Clean → Distorted generative model to synthesize realistic training data for HDR reconstruction.
• To enable high-quality HDR reconstruction, including re-lighting and super-resolution, by combining denoising, deblurring, and spatial/temporal super-resolution.

Proposed method

• The method learns a generative distortion model that maps clean HDR images to realistic sensor readings with correlated pixel noise, row/column noise, and alternating motion blur.
• It uses simple histogram-based statistics to model complex, non-parametric noise distributions from a small number of clean-distorted pairs.
• To simulate exposure-dependent motion blur, it leverages high-speed LDR video to synthesize column-alternating blur patterns that match the sensor's behavior.
• The network is trained in an end-to-end fashion using this synthetic data, enabling robust denoising and deblurring even with limited real HDR data.
• The framework supports both temporal and spatial super-resolution by extending the output to multiple frames or larger patches during training.
• It enables re-lighting applications by producing noise-free, sharp HDR images suitable for path tracing and tone mapping.

Experimental results

Research questions

• RQ1 Can a generative distortion model (Clean → Distorted) effectively replace the need for large-scale real Distorted → Clean paired data in HDR reconstruction?
• RQ2 Can synthetic training data with realistic noise and motion blur patterns from limited clean HDR samples achieve state-of-the-art HDR denoising and deblurring?
• RQ3 Does learning from high-speed LDR video enable accurate simulation of column-alternating motion blur in dual-exposure sensors?
• RQ4 Can the proposed method improve the performance of existing denoisers like BM3D when retrained on its synthetic data?
• RQ5 To what extent can the method support advanced applications such as HDR re-lighting and super-resolution?

Key findings

• The proposed method achieves state-of-the-art HDR reconstruction quality, outperforming existing baselines in both denoising and deblurring under extreme exposure ratios (e.g., 16:1).
• By training on synthetic data generated via the Clean → Distorted model, the method reduces reconstruction error to 0.026 in temporal super-resolution and 0.035 in spatial super-resolution, outperforming standalone methods.
• When retrained on the synthetic data, existing methods like BM3D show significant performance gains, demonstrating the data's generalization and utility.
• The method successfully removes row and column noise while preserving sharp shadows and bright highlights, enabling accurate re-lighting in VR/AR applications as shown in path-traced renderings.
• The use of high-speed LDR video enables effective simulation of exposure-dependent motion blur, which is critical for realistic distortion modeling in dual-exposure sensors.
• End-to-end training with joint denoising, deblurring, and super-resolution leads to superior results compared to sequential application of individual components.