[Paper Review] Transfer Learning from Synthetic to Real-Noise Denoising with Adaptive Instance Normalization

Yoonsik Kim, Jae Woong Soh+2 more|arXiv (Cornell University)|Feb 26, 2020

Image and Signal Denoising Methods56 references20 citations

TL;DR

This paper proposes a transfer learning framework that leverages synthetic-noise data to train a denoiser with adaptive instance normalization (AIN), enabling strong generalization to real-world noise. By fine-tuning the AIN module on a small amount of real-noise data, the method achieves state-of-the-art performance on real-world benchmarks like Darmstadt Noise Dataset (DND), outperforming existing methods even with minimal real data.

ABSTRACT

Real-noise denoising is a challenging task because the statistics of real-noise do not follow the normal distribution, and they are also spatially and temporally changing. In order to cope with various and complex real-noise, we propose a well-generalized denoising architecture and a transfer learning scheme. Specifically, we adopt an adaptive instance normalization to build a denoiser, which can regularize the feature map and prevent the network from overfitting to the training set. We also introduce a transfer learning scheme that transfers knowledge learned from synthetic-noise data to the real-noise denoiser. From the proposed transfer learning, the synthetic-noise denoiser can learn general features from various synthetic-noise data, and the real-noise denoiser can learn the real-noise characteristics from real data. From the experiments, we find that the proposed denoising method has great generalization ability, such that our network trained with synthetic-noise achieves the best performance for Darmstadt Noise Dataset (DND) among the methods from published papers. We can also see that the proposed transfer learning scheme robustly works for real-noise images through the learning with a very small number of labeled data.

Motivation & Objective

• Address the domain gap between synthetic noise (SN) and real-world noise (RN) in image denoising, where CNNs trained on SN fail on RN due to distribution mismatch.
• Overcome overfitting to synthetic noise distributions by introducing regularization via adaptive instance normalization (AIN) to improve generalization.
• Develop a transfer learning scheme that efficiently adapts a SN-trained denoiser to RN by fine-tuning only the AIN module with a small number of real-noise pairs.
• Demonstrate that the proposed method generalizes well across diverse camera devices and noise characteristics, even with limited real data.
• Achieve state-of-the-art performance on real-world denoising benchmarks like DND and SIDD, outperforming prior methods trained on full real datasets.

Proposed method

• Propose AINDNet, a denoising network that uses adaptive instance normalization (AIN) to regularize feature maps and prevent overfitting to synthetic noise.
• Train the AINDNet(S) variant on synthetic-noise data using AIN, where normalization parameters are predicted per spatial location based on local noise level.
• Introduce a transfer learning scheme that fine-tunes only the AIN module of the SN-trained network using a small number of real-noise image pairs.
• Use a noise-level map as input to the AIN module, enabling spatially adaptive normalization that adapts to local noise characteristics.
• Apply the transfer learning process by updating only the affine parameters of AIN layers using real data, preserving the main network weights.
• Train the model end-to-end with L1 or L2 loss on both synthetic and real data, with the real data used only for fine-tuning the AIN module.

(a) Synthetic Data

Experimental results

Research questions

• RQ1 Can a denoiser trained on synthetic noise generalize effectively to real-world noise, despite distributional differences?
• RQ2 How effective is adaptive instance normalization (AIN) in regularizing the network and improving generalization to real noise?
• RQ3 To what extent can transfer learning from synthetic to real noise reduce the need for large-scale real-noise datasets?
• RQ4 What is the performance gain of the proposed transfer learning scheme when only a few real-noise pairs are available?
• RQ5 Does fine-tuning only the AIN module preserve the generalization capacity of the pre-trained synthetic denoiser while adapting to real noise?

Key findings

• The proposed AINDNet(S) model achieves a PSNR of 39.53 dB on the DND test set when trained solely on synthetic noise, outperforming prior state-of-the-art methods.
• With only 32 real-noise pairs from SIDD, the transfer learning variant AINDNet+TF achieves a PSNR of 38.75 dB, surpassing RIDNet trained on the full 320-image real dataset.
• The AINDNet+TF model achieves a PSNR of 39.08 dB on the SIDD validation set, outperforming the best reported method (AINDNet+RT at 39.15 dB) when using the same training data.
• The transfer learning scheme enables significant performance gains even with just 1–2 real images, showing robustness to data scarcity.
• Ablation studies confirm that the AIN-ResBlock architecture outperforms standard IN and gated residual blocks, especially on real-world noise datasets.
• The method generalizes well across diverse camera devices, as evidenced by strong performance on multiple real-world benchmarks including DND and SIDD.

(b) Real Data