[Paper Review] Noise2Self: Blind Denoising by Self-Supervision

Joshua Batson, Löıc A. Royer|arXiv (Cornell University)|Jan 30, 2019

Cell Image Analysis Techniques120 citations

TL;DR

The paper presents Noise2Self, a self-supervised, J-invariant denoising method that learns to denoise without clean targets by masking parts of the input and predicting masked values, supported by Gaussian-process theory and practical UNet-based experiments.

ABSTRACT

We propose a general framework for denoising high-dimensional measurements which requires no prior on the signal, no estimate of the noise, and no clean training data. The only assumption is that the noise exhibits statistical independence across different dimensions of the measurement, while the true signal exhibits some correlation. For a broad class of functions ("$\mathcal{J}$-invariant"), it is then possible to estimate the performance of a denoiser from noisy data alone. This allows us to calibrate $\mathcal{J}$-invariant versions of any parameterised denoising algorithm, from the single hyperparameter of a median filter to the millions of weights of a deep neural network. We demonstrate this on natural image and microscopy data, where we exploit noise independence between pixels, and on single-cell gene expression data, where we exploit independence between detections of individual molecules. This framework generalizes recent work on training neural nets from noisy images and on cross-validation for matrix factorization.

Motivation & Objective

• Motivate and develop a denoising method that does not require clean ground truth data.
• Provide theoretical guarantees comparing self-supervised predictors to Gaussian counterparts.
• Propose practical masking schemes and trainable architectures to realize J-invariant denoisers.
• Demonstrate effectiveness on diverse datasets using neural networks and calibration analyses.

Proposed method

• Formulate J-invariant self-supervision where the predictor uses masked/interpolated inputs to predict masked coordinates.
• Introduce g(x)_J := f_theta(1_J^c * x_J + 1_J * s(x))_J, with s(x) as a local interpolation (e.g., a fixed kernel).
• Present Gaussian-process based lemmas and propositions showing that, under Gaussian assumptions, the optimal J-invariant predictor is no worse than the Gaussian case and that conditioning on x_J^c yields a bound on the MSE.
• Discuss masking strategies (Uniform Pixel Selection) and the issue of information leakage, advocating truly J-invariant schemes where the masked value does not influence the prediction.
• Propose and analyze linear combinations of uncorrelated unbiased estimators to potentially improve PSNR when combining the self-supervised predictor with the original noisy input.
• Describe training with a randomized J (subset of coordinates) and a UNet architecture, plus inference options with true J-invariance or direct full-input application.

Experimental results

Research questions

• RQ1 Can denoising be effectively learned without clean targets using a J-invariant self-supervised setup?
• RQ2 How does the self-supervised predictor compare to Gaussian baselines in terms of mean-squared error?
• RQ3 What masking/interpolation strategies best realize true J-invariance and maximize denoising performance?
• RQ4 Is it beneficial to linearly combine the self-supervised predictor with the original noisy input, and how does this affect PSNR?

Key findings

• A Gaussian-process based result shows that the self-supervised J-invariant predictor is competitive with or better than the corresponding Gaussian predictor. (Proposition 1)
• Under certain conditions, combining the self-supervised estimator with the raw input using an optimal linear mix can reduce variance and potentially improve PSNR. (Section 3.2)
• Uniform Pixel Selection masking can bias the predictor unless care is taken to ensure J-invariance; the authors propose excluding the value being predicted when estimating local distributions.
• Training with randomized J and a UNet can produce genuinely J-invariant predictions, and inference with full input can yield about 0.5 dB PSNR improvement over the strictly J-invariant version on the evaluated datasets.
• The authors calibrate traditional denoisers (median, wavelet, NL-means) using self-supervised loss to select optimal parameters without ground-truth references (calibration plots).
• Experiments on Hànzì, CellNet, and ImageNet with a UNet demonstrate practical self-supervised denoising across synthetic and real-noise models.