[Paper Review] VIDOSAT: High-dimensional Sparsifying Transform Learning for Online Video Denoising

Bihan Wen, Saiprasad Ravishankar+1 more|arXiv (Cornell University)|Oct 3, 2017

Image and Signal Denoising Methods40 references3 citations

TL;DR

This paper proposes VIDOSAT, an online video denoising framework that learns high-dimensional sparsifying transforms from spatio-temporal video patches in real time. By leveraging online transform learning and optional block matching for motion compensation, VIDOSAT achieves state-of-the-art denoising performance, outperforming VBM3D, VBM4D, and fixed 3D DCT by up to 3.5 dB in average PSNR across multiple datasets.

ABSTRACT

Techniques exploiting the sparsity of images in a transform domain have been effective for various applications in image and video processing. Transform learning methods involve cheap computations and have been demonstrated to perform well in applications such as image denoising and medical image reconstruction. Recently, we proposed methods for online learning of sparsifying transforms from streaming signals, which enjoy good convergence guarantees, and involve lower computational costs than online synthesis dictionary learning. In this work, we apply online transform learning to video denoising. We present a novel framework for online video denoising based on high-dimensional sparsifying transform learning for spatio-temporal patches. The patches are constructed either from corresponding 2D patches in successive frames or using an online block matching technique. The proposed online video denoising requires little memory, and offers efficient processing. Numerical experiments compare the performance to the proposed video denoising scheme but fixing the transform to be 3D DCT, as well as prior schemes such as dictionary learning-based schemes, and the state-of-the-art VBM3D and VBM4D on several video data sets, demonstrating the promising performance of the proposed methods.

Motivation & Objective

• Address the challenge of real-time, high-quality video denoising in dynamic scenes with complex motion.
• Overcome limitations of fixed transforms (e.g., 3D DCT) and expensive synthesis dictionary learning in video restoration.
• Develop an efficient, online learning framework that adapts sparsifying transforms to evolving video content.
• Improve denoising performance by modeling spatio-temporal correlations through adaptive transform learning.
• Enable memory-efficient, scalable video denoising suitable for big data and streaming applications.

Proposed method

• Construct spatio-temporal patches from consecutive video frames or via online block matching to capture motion.
• Learn high-dimensional sparsifying transforms (W) in an online fashion using streaming video data.
• Use thresholding of Wu to compute the sparse code x efficiently, avoiding NP-hard synthesis coding.
• Update the transform W iteratively using a stochastic gradient descent approach with convergence guarantees.
• Integrate block matching into the learning pipeline (VIDOSAT-BM) to group similar patches across time and space.
• Apply the learned transform to denoise each patch and reconstruct the video frame by frame.

Experimental results

Research questions

• RQ1 Can online sparsifying transform learning improve video denoising performance compared to fixed transforms like 3D DCT?
• RQ2 How does online transform learning compare to dictionary learning-based methods in terms of speed, accuracy, and memory usage?
• RQ3 To what extent does incorporating block matching into the transform learning pipeline enhance denoising for videos with motion?
• RQ4 Can the proposed method achieve real-time processing with low memory footprint while maintaining high PSNR?
• RQ5 How do the learned transforms adapt to dynamic video content over time?

Key findings

• VIDOSAT-BM achieved average PSNR improvements of 3.5 dB over VBM3D, 2.1 dB over VBM4D, and 1.2 dB over sKSVD across all test videos and noise levels.
• VIDOSAT-BM consistently outperformed all competing methods, including VBM3D and VBM4D, for every video and noise level.
• The average PSNR of VIDOSAT-BM was 0.3 dB higher than VIDOSAT, demonstrating the benefit of block matching for motion modeling.
• The learned transforms in VIDOSAT evolved over time, showing adaptive structures such as linear shifts along depth that compensate for motion.
• In highly dynamic videos like Bicycle, VIDOSAT-BM significantly outperformed VIDOSAT and VBM4D, with stable frame-by-frame PSNR improvements.
• Visual results confirmed that VIDOSAT-BM reduced artifacts in moving regions while preserving detail in static areas, surpassing VBM4D and VBM3D in quality.