[Paper Review] Removing the Feature Correlation Effect of Multiplicative Noise

Zijun Zhang, Yining Zhang+1 more|arXiv (Cornell University)|Sep 19, 2018

Domain Adaptation and Few-Shot Learning3 citations

TL;DR

This paper proposes Non-Correlating Multiplicative Noise (NCMN), a novel regularization technique that removes the feature correlation induced by standard multiplicative noise like dropout in batch-normalized deep networks. By truncating gradients through the noise component and combining it with batch normalization, NCMN reduces feature correlation, improves generalization, and outperforms both standard dropout and shake-shake regularization on image classification benchmarks.

ABSTRACT

Multiplicative noise, including dropout, is widely used to regularize deep neural networks (DNNs), and is shown to be effective in a wide range of architectures and tasks. From an information perspective, we consider injecting multiplicative noise into a DNN as training the network to solve the task with noisy information pathways, which leads to the observation that multiplicative noise tends to increase the correlation between features, so as to increase the signal-to-noise ratio of information pathways. However, high feature correlation is undesirable, as it increases redundancy in representations. In this work, we propose non-correlating multiplicative noise (NCMN), which exploits batch normalization to remove the correlation effect in a simple yet effective way. We show that NCMN significantly improves the performance of standard multiplicative noise on image classification tasks, providing a better alternative to dropout for batch-normalized networks. Additionally, we present a unified view of NCMN and shake-shake regularization, which explains the performance gain of the latter.

Motivation & Objective

• To identify and address the previously overlooked side effect of multiplicative noise—increased feature correlation—which undermines generalization.
• To develop a regularization method that removes this correlation effect without introducing significant computational overhead or extra hyperparameters.
• To provide a unified theoretical explanation for the success of shake-shake regularization through its similarity to a variant of NCMN.
• To demonstrate that NCMN consistently improves performance across diverse architectures, especially ResNets and wide residual networks.

Proposed method

• Decompose noisy features into signal and noise components, then truncate gradients through the noise component to prevent optimization drift.
• Introduce a gradient modification that treats the noise component as constant during backpropagation, effectively removing its influence on feature updates.
• Combine the gradient-truncation technique with batch normalization to counteract the tendency of feature magnitudes to grow during training.
• Formulate multiple variants of NCMN (NCMN-0, NCMN-1, NCMN-2) tailored for different network architectures, including plain CNNs and residual networks.
• Theoretically link NCMN to shake-shake regularization by showing that both methods produce noise with similar statistical properties that suppress feature correlation.
• Apply the method to fully connected and convolutional layers, validating its effectiveness in standard image classification settings.

Experimental results

Research questions

• RQ1 Does multiplicative noise like dropout inherently increase feature correlation, and if so, how does this affect model generalization?
• RQ2 Can the feature correlation effect of multiplicative noise be systematically removed without introducing additional hyperparameters or computational cost?
• RQ3 What is the theoretical connection between NCMN and shake-shake regularization, and does it explain shake-shake’s superior performance?
• RQ4 How does NCMN compare to standard dropout and shake-shake in terms of test accuracy and training stability across different network architectures?
• RQ5 Does the benefit of NCMN extend to non-CNN architectures such as LSTMs, where feature correlation is already low?

Key findings

• NCMN significantly reduces feature correlation, achieving lower correlation levels than both clean features and those with standard multiplicative noise.
• On CIFAR-10 and CIFAR-100, NCMN-2 achieves a test error of 3.00% on WRN-22-7.5, outperforming standard dropout (3.89%) and shake-shake (3.51%) with the same architecture.
• A WRN-28-10 model trained with NCMN-2 reaches comparable performance to shake-shake in 9 times fewer epochs (200 vs. 1800), demonstrating faster convergence.
• NCMN-0 provides consistent performance gains across models of varying sizes, including small (1.6M params) and large (36.5M params) networks, with improvements up to 5.10% on CIFAR-10.
• NCMN-2 yields the best generalization on ResNets, while NCMN-0 is simpler and faster for general use, showing architectural sensitivity in performance.
• In LSTMs, where feature correlation is already low, NCMN provides no significant improvement over dropout, confirming that the benefit is most pronounced in high-correlation settings.