[Paper Review] Noise2Noise: Learning Image Restoration without Clean Data

Jaakko Lehtinen, Jacob Munkberg+5 more|arXiv (Cornell University)|Mar 12, 2018

Image and Signal Denoising Methods982 citations

TL;DR

The paper shows that neural networks can learn to restore images using only corrupted (noisy) examples, without requiring clean targets or explicit noise models, achieving comparable or better results than training with clean data across multiple tasks.

ABSTRACT

We apply basic statistical reasoning to signal reconstruction by machine learning -- learning to map corrupted observations to clean signals -- with a simple and powerful conclusion: it is possible to learn to restore images by only looking at corrupted examples, at performance at and sometimes exceeding training using clean data, without explicit image priors or likelihood models of the corruption. In practice, we show that a single model learns photographic noise removal, denoising synthetic Monte Carlo images, and reconstruction of undersampled MRI scans -- all corrupted by different processes -- based on noisy data only.

Motivation & Objective

• Demonstrate that image restoration can be learned from corrupted data alone, without explicit noise models or clean targets.
• Show that training with noisy targets yields performance comparable to training with clean targets across diverse tasks.
• Illustrate practical implications for data collection by highlighting benefits under finite data and varied noise processes.
• Explore multiple noise types (Gaussian, Poisson, Bernoulli) and inverse problems (MRI reconstruction) to validate the approach.

Proposed method

• Train neural network regressors to minimize L(f_theta(x_hat), y_hat) using corrupted inputs x_hat and corrupted targets y_hat, where E[y_hat|x_hat] equals the clean target y.
• Use the fact that, under L2 loss, the optimal predictor equals the conditional expectation, allowing clean targets to be replaced by zero-mean corrupted targets without changing the learned mapping.
• Demonstrate that with Gaussian, Poisson, and Bernoulli/noise corruptions, noisy targets yield similar or better restoration performance compared to clean targets.
• For Monte Carlo rendering and MRI, adapt loss functions and input representations to account for noise distributions and data fidelity, including HDR-aware losses and spectrum-preserving strategies.

Experimental results

Research questions

• RQ1 Can neural networks learn to restore images when only corrupted pairs are available, with no clean targets?
• RQ2 Do different noise models (Gaussian, Poisson, Bernoulli, impulse) affect the effectiveness of noisy-target training?
• RQ3 Is noisy-target training effective for inverse problems like undersampled MRI reconstruction and Monte Carlo rendered images?
• RQ4 How does finite data and capture budget influence the performance of noisy-target training compared to clean-target training?
• RQ5 What loss functions and input representations best support noisy-target training under challenging corruptions?

Key findings

• Noisy-target training achieves denoising performance close to or matching clean-target training across Gaussian, Poisson, and Bernoulli noise on Kodak, BSD300, and Set14 datasets.
• Corrupted targets can offer benefits under finite data by exposing the model to more realizations of the corruption and more latent clean images, sometimes improving results over clean targets.
• For Monte Carlo rendering, training with noisy targets yields PSNR within ~0.5 dB of clean-target training, while being thousands of times faster to generate data.
• In MRI, training with noisy targets achieves PSNR comparable to clean-target training for undersampled k-space reconstruction, validating the approach for inverse problems.
• Different losses (L2, L1, L0-like mode seeking) enable robust restoration under various corruption regimes, with L0-based losses offering advantages for heavy impulse noise.
• Online training experiments show that noisy-target denoisers trained in real-time can approach the quality of models trained with clean data, with substantially faster data generation.