[Paper Review] Supervised Raw Video Denoising with a Benchmark Dataset on Dynamic Scenes

Huanjing Yue, Cong Cao+3 more|arXiv (Cornell University)|Mar 31, 2020

Image and Signal Denoising Methods45 references7 citations

TL;DR

This paper proposes RViDeNet, a supervised raw video denoising method that leverages a novel benchmark dataset of 55 groups of dynamic noisy-clean video pairs captured at ISO 1600–25600. By decomposing Bayer-patterned raw video into RGBG sub-sequences and exploiting spatial, channel, and temporal correlations via non-local attention and temporal fusion, the method achieves state-of-the-art performance in both raw and sRGB domains, outperforming existing video and image denoising models by up to 1.12 dB in PSNR.

ABSTRACT

In recent years, the supervised learning strategy for real noisy image denoising has been emerging and has achieved promising results. In contrast, realistic noise removal for raw noisy videos is rarely studied due to the lack of noisy-clean pairs for dynamic scenes. Clean video frames for dynamic scenes cannot be captured with a long-exposure shutter or averaging multi-shots as was done for static images. In this paper, we solve this problem by creating motions for controllable objects, such as toys, and capturing each static moment for multiple times to generate clean video frames. In this way, we construct a dataset with 55 groups of noisy-clean videos with ISO values ranging from 1600 to 25600. To our knowledge, this is the first dynamic video dataset with noisy-clean pairs. Correspondingly, we propose a raw video denoising network (RViDeNet) by exploring the temporal, spatial, and channel correlations of video frames. Since the raw video has Bayer patterns, we pack it into four sub-sequences, i.e RGBG sequences, which are denoised by the proposed RViDeNet separately and finally fused into a clean video. In addition, our network not only outputs a raw denoising result, but also the sRGB result by going through an image signal processing (ISP) module, which enables users to generate the sRGB result with their favourite ISPs. Experimental results demonstrate that our method outperforms state-of-the-art video and raw image denoising algorithms on both indoor and outdoor videos.

Motivation & Objective

• To address the lack of real noisy-clean video pairs for dynamic scenes, which hinders supervised learning in raw video denoising.
• To develop a method that effectively exploits temporal, spatial, and channel correlations in raw video for superior denoising performance.
• To enable flexible output by generating both raw denoised and sRGB results via an integrated ISP module.
• To establish a reproducible and quantitatively evaluable benchmark for future research in raw video denoising.

Proposed method

• The authors construct a benchmark dataset by manually creating controlled motion for objects and capturing multiple high-ISO exposures per static moment, then averaging them to form clean frames.
• Raw video is split into four RGBG sub-sequences to preserve Bayer pattern structure and enable per-sequence processing.
• The RViDeNet architecture applies pre-denoising, frame alignment, non-local attention, and temporal fusion modules to exploit spatiotemporal correlations across frames.
• Non-local attention mechanisms are applied across spatial, channel, and temporal dimensions to model long-range dependencies in the raw video data.
• The denoised RGBG sequences are spatially fused and passed through a learnable ISP module to generate sRGB outputs, allowing user customization.
• The network is trained end-to-end on real noisy-clean video pairs, with both raw and sRGB supervision, enabling joint optimization of denoising and color fidelity.

Experimental results

Research questions

• RQ1 Can a supervised learning approach achieve superior video denoising performance when trained on real noisy-clean video pairs for dynamic scenes?
• RQ2 How effectively can spatiotemporal and channel-wise correlations in raw video be exploited to improve denoising beyond single-frame or sRGB-based methods?
• RQ3 To what extent does the proposed RViDeNet outperform state-of-the-art video and raw image denoising models in both quantitative and qualitative metrics?
• RQ4 Can the integration of a flexible ISP module enable users to generate preferred sRGB outputs without retraining?
• RQ5 How generalizable is the model when fine-tuned on synthetic data, and does it maintain performance on real-world low-light scenes?

Key findings

• On indoor videos with ground truth, RViDeNet achieves a PSNR of 43.97 and SSIM of 0.9874 in the raw domain, outperforming the next-best method (DIDN) by 0.72 dB in PSNR.
• In the sRGB domain, RViDeNet achieves a PSNR of 39.95 and SSIM of 0.9792, surpassing DIDN by 1.12 dB in PSNR and EDVR by nearly 1 dB.
• The ablation study shows that the full RViDeNet model with real data training significantly outperforms the variant trained only on synthetic data (denoted Ours⁻), indicating effective generalization.
• Visual comparisons demonstrate that RViDeNet removes noise more effectively while preserving fine details and maintaining temporal consistency, unlike over-smoothed results from EDVR and DIDN.
• User studies on outdoor videos without ground truth confirm that RViDeNet produces more visually pleasing results with better texture recovery and reduced flicker.
• The proposed dataset of 55 dynamic video sequences with ISO 1600–25600 provides the first real-world benchmark for supervised raw video denoising, enabling reproducible evaluation.