[Paper Review] Implementation of the VBM3D Video Denoising Method and Some Variants

Thibaud Ehret, Pablo Arias|arXiv (Cornell University)|Jan 6, 2020

Image and Signal Denoising Methods22 references4 citations

TL;DR

This paper presents an open-source C++ implementation of the VBM3D video denoising algorithm, an extension of BM3D for video that groups similar spatio-temporal patches into 3D stacks for joint denoising via 3D transforms and shrinkage. The key contribution is demonstrating that combining 3D patches, optical flow-guided patch search, and multi-scale processing yields state-of-the-art performance, especially at high noise levels (σ=40), with PSNR improvements up to 3.5 dB over baseline VBM3D.

ABSTRACT

VBM3D is an extension to video of the well known image denoising algorithm BM3D, which takes advantage of the sparse representation of stacks of similar patches in a transform domain. The extension is rather straightforward: the similar 2D patches are taken from a spatio-temporal neighborhood which includes neighboring frames. In spite of its simplicity, the algorithm offers a good trade-off between denoising performance and computational complexity. In this work we revisit this method, providing an open-source C++ implementation reproducing the results. A detailed description is given and the choice of parameters is thoroughly discussed. Furthermore, we discuss several extensions of the original algorithm: (1) a multi-scale implementation, (2) the use of 3D patches, (3) the use of optical flow to guide the patch search. These extensions allow to obtain results which are competitive with even the most recent state of the art.

Motivation & Objective

• To provide a reproducible, open-source C++ implementation of the VBM3D video denoising algorithm.
• To analyze and validate the performance of VBM3D variants, including multi-scale processing, 3D patch usage, and optical flow-guided patch search.
• To demonstrate that these extensions can achieve state-of-the-art denoising performance, particularly under high noise conditions.
• To offer a detailed parameter analysis and implementation guidance for researchers and practitioners.

Proposed method

• The algorithm extends BM3D to video by grouping similar 2D patches from a spatio-temporal neighborhood into 3D stacks.
• A separable 3D transform (e.g., DCT or wavelet) is applied to each 3D stack to concentrate signal energy in a few coefficients.
• Two-stage denoising is applied: first a hard thresholding step using a 3D shrinkage operator, followed by a Wiener filtering step using the first estimate as an oracle.
• Patch search is guided by optical flow to improve similarity detection in motion-compensated regions.
• Multi-scale processing is applied by denoising at multiple resolutions and fusing results, with interpolation using Lanczos or DCT kernels.
• Final video reconstruction uses patch aggregation, where each pixel is estimated as a weighted average of overlapping denoised patches.

Experimental results

Research questions

• RQ1 Can a straightforward extension of BM3D to video (VBM3D) achieve competitive denoising performance with low computational overhead?
• RQ2 How does using 3D spatio-temporal patches instead of 2D patches affect denoising quality and robustness to motion?
• RQ3 To what extent does optical flow guidance improve patch similarity detection and overall denoising performance?
• RQ4 Does multi-scale processing enhance denoising quality, or does it introduce artifacts and degrade performance?
• RQ5 How do the proposed variants compare to the latest state-of-the-art video denoising methods in terms of PSNR and visual quality?

Key findings

• The proposed VBM3D implementation with optical flow-guided patch search and multi-scale processing achieves a PSNR of 38.95 dB on the Derf test set at σ=10, outperforming the original VBM3D (38.10 dB) and matching or exceeding SOTA methods like VNLB and VNLnet.
• At σ=40, the best variant (VBM3D ST+OF+MS with Lanczos interpolation) achieves 32.44 dB PSNR, significantly outperforming the original VBM3D (30.96 dB) and approaching the performance of VNLB (32.95 dB).
• The use of 3D patches and optical flow guidance leads to substantial PSNR gains, especially in high-motion or complex scenes, with improvements of up to 1.5 dB over standard VBM3D.
• Multi-scale processing does not consistently improve performance; in some cases, it degrades results, suggesting trade-offs between resolution and noise suppression.
• Visual comparisons confirm that the proposed variants reduce ringing and blurring artifacts better than baseline VBM3D, especially in textured and motion-rich regions.
• The open-source C++ implementation is shown to reproduce the results reported in the original VBM3D paper, ensuring reproducibility and facilitating further research.