[Paper Review] Learning Deformable Kernels for Image and Video Denoising

Xiangyu Xu, Muchen Li+1 more|arXiv (Cornell University)|Apr 15, 2019

Image and Signal Denoising Methods30 references37 citations

TL;DR

The paper proposes deformable 2D and 3D kernels learned via a CNN to explicitly perform pixel sampling and weighting for image and video denoising, improving handling of structure, misalignment, and large motion.

ABSTRACT

Most of the classical denoising methods restore clear results by selecting and averaging pixels in the noisy input. Instead of relying on hand-crafted selecting and averaging strategies, we propose to explicitly learn this process with deep neural networks. Specifically, we propose deformable 2D kernels for image denoising where the sampling locations and kernel weights are both learned. The proposed kernel naturally adapts to image structures and could effectively reduce the oversmoothing artifacts. Furthermore, we develop 3D deformable kernels for video denoising to more efficiently sample pixels across the spatial-temporal space. Our method is able to solve the misalignment issues of large motion from dynamic scenes. For better training our video denoising model, we introduce the trilinear sampler and a new regularization term. We demonstrate that the proposed method performs favorably against the state-of-the-art image and video denoising approaches on both synthetic and real-world data.

Motivation & Objective

• Motivate learning the classical selecting and averaging process in denoising rather than relying on hand-crafted rules.
• Develop deformable kernels that adapt sampling locations and weights to image structure for improved denoising quality.
• Extend to 3D deformable kernels to handle large motion in video denoising and reduce artifacts.
• Introduce training strategies including an annealing-based regularization to stabilize 3D kernel learning.
• Evaluate against state-of-the-art image and video denoising methods on synthetic and real data.

Proposed method

• Estimate per-pixel deformable kernel offsets with a CNN (offset network).
• Sample pixels from the noisy input at predicted offset locations using bilinear/trilinear interpolation.
• Compute spatially-variant kernel weights F from concatenated sampled pixels, input, and offset features to form Y = sum X(sampled) * F.
• For video, use 3D deformable kernels that sample across spatial-temporal space with trilinear interpolation.
• Introduce an annealing regularization that encourages diverse temporal sampling across kernel groups during training.
• Train end-to-end in linear space with optional gamma correction for perceptual quality.

Experimental results

Research questions

• RQ1 Can deformable, learnable sampling grids outperform fixed rigid kernels for image and video denoising?
• RQ2 Do 3D deformable kernels better handle large motion and misalignment in dynamic scenes compared to 2D approaches?
• RQ3 Does an annealing-based regularization improve optimization and temporal utilization of samples in video denoising?
• RQ4 How do learned deformable kernels compare to state-of-the-art denoising methods on synthetic and real data?

Key findings

• The deformable kernels learn to adapt sampling locations and weights, improving denoising quality and reducing artifacts compared to rigid kernels.
• 3D deformable kernels outperform 2D counterparts on videos with large motion by distributing samples across reliable frames.
• The proposed method achieves favorable PSNR/SSIM against state-of-the-art methods on both single-image and video denoising in synthetic experiments.
• An annealing regularization helps avoid local minima that concentrate samples around the reference frame, enabling broader temporal utilization.
• The approach generalizes to real noisy images and cellphone-captured videos, recovering subtle edges and reducing artifacts.
• Ablation studies show the importance of dynamic weights, deformable sampling, and annealing for performance.