[Paper Review] Fully Convolutional Pixel Adaptive Image Denoiser

Sungmin Cha, Taesup Moon|arXiv (Cornell University)|Jul 19, 2018

Image and Signal Denoising Methods38 references4 citations

TL;DR

This paper proposes a fully convolutional, pixel-adaptive image denoising network (FC-AIDE) that leverages a learnable polynomial mapping per pixel to adaptively adjust denoising strength based on local image content. By deriving an unbiased estimator of the true mean squared error using Stein's Unbiased Risk Estimate (SURE), the method enables end-to-end training with a differentiable loss, achieving state-of-the-art performance on multiple benchmark datasets with significant PSNR gains over baselines.

ABSTRACT

We propose a new image denoising algorithm, dubbed as Fully Convolutional Adaptive Image DEnoiser (FC-AIDE), that can learn from an offline supervised training set with a fully convolutional neural network as well as adaptively fine-tune the supervised model for each given noisy image. We significantly extend the framework of the recently proposed Neural AIDE, which formulates the denoiser to be context-based pixelwise mappings and utilizes the unbiased estimator of MSE for such denoisers. The two main contributions we make are; 1) implementing a novel fully convolutional architecture that boosts the base supervised model, and 2) introducing regularization methods for the adaptive fine-tuning such that a stronger and more robust adaptivity can be attained. As a result, FC-AIDE is shown to possess many desirable features; it outperforms the recent CNN-based state-of-the-art denoisers on all of the benchmark datasets we tested, and gets particularly strong for various challenging scenarios, e.g., with mismatched image/noise characteristics or with scarce supervised training data. The source code of our algorithm is available at https://github.com/csm9493/FC-AIDE-Keras.

Motivation & Objective

• To develop a fully convolutional, pixel-adaptive image denoising method that adjusts denoising strength per pixel based on local image content.
• To address the limitation of fixed-denoising-strength models by introducing learnable polynomial coefficients for adaptive noise handling.
• To enable end-to-end training via a differentiable, unbiased estimator of the true MSE using SURE, improving generalization and performance.
• To demonstrate superior denoising performance across diverse datasets including BSD68, Set12, Urban100, and medical images.

Proposed method

• The method uses a fully convolutional neural network with per-pixel polynomial mappings of degree d ∈ {1,2} to adaptively adjust denoising strength based on local image statistics.
• It introduces a novel loss function derived from Stein's Unbiased Risk Estimate (SURE), which provides an unbiased estimate of the true MSE without requiring ground truth.
• The loss function is designed to be differentiable and enables end-to-end training by estimating the trace of the Jacobian via a closed-form expression.
• The model is trained in two stages: supervised pre-training (FC-AIDE_S) followed by adaptive fine-tuning (FC-AIDE_S+FT) using the SURE-based loss.
• Self-ensemble data augmentation is applied during fine-tuning to improve robustness and performance.
• Hyperparameters for ℓ₂-Sparse Regularization (ℓ₂-SP) and early stopping are selected via validation on a separate set of images for each noise level.

Experimental results

Research questions

• RQ1 Can a fully convolutional, pixel-adaptive denoising network achieve better performance than fixed-strength denoisers by learning spatially varying denoising coefficients?
• RQ2 Does the proposed SURE-based loss function provide an unbiased and effective surrogate for the true MSE during end-to-end training?
• RQ3 How do the individual polynomial coefficients (a₀, a₁, a₂) contribute to denoising performance, and what is the impact of removing them?
• RQ4 What is the optimal strategy for data augmentation and hyperparameter selection in the fine-tuning stage to maximize performance and robustness?
• RQ5 How does the adaptive fine-tuning with the SURE loss compare to standard end-to-end training in terms of PSNR and convergence stability?

Key findings

• FC-AIDE_S+FT achieves a PSNR of 29.31 dB on BSD68 at σ=25, significantly outperforming FC-AIDE_S (27.66 dB) and the ablated model FC-AIDE_S+FT(d=2, a₀-only) (19.20 dB).
• The ablation study confirms that the a₁ and a₂ coefficients are essential: removing them reduces performance to near-noise level (20.46 dB), demonstrating their critical role in adaptive denoising.
• The SURE-based loss is empirically unbiased, as shown by the histogram of MSE minus the loss value concentrating at zero across 100 noise realizations.
• Adaptive fine-tuning with self-ensemble data augmentation leads to faster convergence and higher PSNR than training with no augmentation or only training/test augmentation.
• The model achieves maximum PSNR improvement within 5 epochs, with early stopping feasible due to stable convergence, and the full fine-tuning scheme outperforms all ablation variants.