[Paper Review] Image denoising with multi-layer perceptrons, part 2: training trade-offs and analysis of their mechanisms

Harold Christopher Burger, Christian J. Schuler+1 more|arXiv (Cornell University)|Nov 7, 2012

Image and Signal Denoising Methods23 references26 citations

TL;DR

This paper investigates training trade-offs and internal mechanisms of multi-layer perceptrons (MLPs) for image denoising, demonstrating that deeper MLPs with proper hyperparameter tuning achieve state-of-the-art performance. It reveals that denoising works via feature detection and saturation in tanh units, producing binary-like representations that act as implicit regularization, with activation pattern analysis enabling interpretability of the model's internal logic.

ABSTRACT

Image denoising can be described as the problem of mapping from a noisy image to a noise-free image. In another paper, we show that multi-layer perceptrons can achieve outstanding image denoising performance for various types of noise (additive white Gaussian noise, mixed Poisson-Gaussian noise, JPEG artifacts, salt-and-pepper noise and noise resembling stripes). In this work we discuss in detail which trade-offs have to be considered during the training procedure. We will show how to achieve good results and which pitfalls to avoid. By analysing the activation patterns of the hidden units we are able to make observations regarding the functioning principle of multi-layer perceptrons trained for image denoising.

Motivation & Objective

• To identify and resolve critical training trade-offs in MLPs for image denoising, especially in large-scale, time-intensive experiments.
• To understand why certain hyperparameter choices lead to catastrophic performance degradation despite initial progress.
• To analyze hidden unit activation patterns to uncover the internal working principles of trained MLPs for image denoising.
• To evaluate the role of architectural depth, patch size, and fine-tuning in achieving optimal denoising performance.
• To compare the functional behavior of single- and multi-layer MLPs and relate their mechanisms to denoising autoencoders and RBMs.

Proposed method

• Training deep MLPs with up to four hidden layers, each with 2047 units, on large datasets of noisy and clean image patches.
• Using stochastic gradient descent with adaptive learning rates and monitoring both training and test PSNR to track performance evolution.
• Applying activation maximization to visualize the input patterns that maximally activate individual hidden units.
• Analyzing weight updates across neighboring patches in block-matching MLPs to assess feature detector consistency.
• Comparing models with varying patch sizes, network depths, and hidden unit counts to identify optimal configurations.
• Employing fine-tuning with reduced learning rates to improve final test performance.

Experimental results

Research questions

• RQ1 Which training hyperparameters and configurations lead to stable convergence and optimal denoising performance in MLPs?
• RQ2 Why do some training runs degrade catastrophically despite initial performance gains?
• RQ3 How do hidden unit activation patterns reveal the internal functioning of MLPs trained for image denoising?
• RQ4 To what extent do deeper MLPs follow the same functional principle as single-layer models?
• RQ5 How does the type and strength of noise influence the learned feature detectors and generators?

Key findings

• Long training times do not cause overfitting; instead, they are essential for convergence, especially in high-capacity models.
• Larger architectures consistently outperform smaller ones, and more training data always improves performance.
• An optimal number of hidden layers exists; exceeding it leads to catastrophic performance degradation.
• Fine-tuning with a lower learning rate produces significant performance gains, especially in deeper networks.
• The hidden units in trained MLPs detect specific image features, and their activation patterns reveal that denoising relies on feature detection and tanh saturation.
• The model’s internal representations are effectively binary due to tanh saturation, supporting the regularization interpretation of denoising autoencoders.