[Paper Review] High-Quality Self-Supervised Deep Image Denoising

Samuli Laine, Tero Karras+2 more|arXiv (Cornell University)|Jan 29, 2019

Image and Signal Denoising Methods38 references34 citations

TL;DR

A self-supervised denoising framework using blind-spot convolutional networks and Bayesian post-processing achieves near supervised-quality results across Gaussian, Poisson, and impulse noise without clean targets or paired data. It also handles unknown noise parameters and variable noise levels.

ABSTRACT

We describe a novel method for training high-quality image denoising models based on unorganized collections of corrupted images. The training does not need access to clean reference images, or explicit pairs of corrupted images, and can thus be applied in situations where such data is unacceptably expensive or impossible to acquire. We build on a recent technique that removes the need for reference data by employing networks with a "blind spot" in the receptive field, and significantly improve two key aspects: image quality and training efficiency. Our result quality is on par with state-of-the-art neural network denoisers in the case of i.i.d. additive Gaussian noise, and not far behind with Poisson and impulse noise. We also successfully handle cases where parameters of the noise model are variable and/or unknown in both training and evaluation data.

Motivation & Objective

• Motivate denoising when clean references are unavailable and data are only noisy images.
• Propose a blind-spot convolutional network architecture that enables self-supervised training without masking penalties.
• Introduce a Bayesian post-processing step to incorporate the noisy center pixel and noise model at test time.
• Demonstrate robustness to unknown or variable noise parameters across multiple noise models (Gaussian, Poisson, impulse).
• Compare against supervised baselines and masking-based self-supervised methods to establish performance trade-offs.

Proposed method

• Use blind-spot networks whose receptive field excludes the center pixel, enabling training with single noisy images.
• Train to model p(x|Ω_y) as a multivariate Gaussian with mean μ_x and covariance Σ_x, predicting these from the context Ω_y.
• Derive a closed-form posterior p(x|y,Ω_y) by combining the noise model p(y|x) with the learned prior p(x|Ω_y) and use the posterior mean for denoising.
• Parameterize Σ_x via an upper-triangular A_x with Σ_x = A_x^T A_x to ensure positive semidefiniteness.
• Handle Gaussian noise where Σ_y = Σ_x + σ^2 I and μ_y = μ_x; optionally learn σ per image or predict it with an auxiliary network.
• Extend to Poisson and impulse noise by appropriate noise-model likelihoods and moment-matching to obtain training losses and posteriors (Eq. 1–10).
• Use a five-level U-Net backbone with three 1x1 layers, trained on large natural-image datasets, and evaluate on Kodak, BSD300, and Set14.

Experimental results

Research questions

• RQ1 Can self-supervised training on unpaired noisy images achieve denoising quality close to supervised methods for common noise models (Gaussian, Poisson, impulse)?
• RQ2 How can a blind-spot architecture be trained efficiently without masking, and how does Bayesian posterior averaging improve denoising performance?
• RQ3 Do unknown or variable noise parameters (σ, λ, α) degrade performance, and can they be learned or estimated during training?
• RQ4 What is the impact of modeling multivariate correlations in the prior (Σ_x) versus simplifying to diagonal covariances?
• RQ5 How does the proposed method compare to Noise2C, Noise2Noise, and masking-based self-supervised approaches in both quality and training efficiency?

Key findings

• For Gaussian noise, the full blind-spot + posterior mean approach nearly matches supervised baselines (Noise2Clean) and Noise2Noise, with multivariate Σ_x outperforming diagonal Σ_x.
• Posterior mean denoising with learned μ_x and Σ_x outperforms μ-only ablations and masking-based training by substantial margins, especially when color channels are modeled jointly.
• The method remains competitive when σ is unknown or estimated, and even with variable σ, the approach maintains high PSNR across datasets, sometimes with minor drops.
• For Poisson noise, the method is within 0.1–0.2 dB of the Noise2Clean baseline under fixed noise and remains robust under unknown λ, though some datasets (e.g., Set14) show larger degradation when estimating parameters.
• For impulse noise, the approach substantially leverages the center-pixel information via Bayesian fusion, outperforming μ-only ablations and masking-based training, though under certain settings it may lag the Noise2Clean baseline slightly.
• Overall, the proposed self-supervised scheme can reach denoising quality close to supervised models across diverse noise models, with advantages in training efficiency and data requirements.