[Paper Review] The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent

Karthik Abinav Sankararaman, Soham De+3 more|arXiv (Cornell University)|Apr 15, 2019

Stochastic Gradient Optimization TechniquesComputer Science72 references35 citations

TL;DR

The paper defines gradient confusion to analyze SGD dynamics in over-parameterized networks and shows width reduces confusion while depth increases it; techniques like batch normalization and skip connections mitigate the depth-induced training burden.

ABSTRACT

This paper studies how neural network architecture affects the speed of training. We introduce a simple concept called gradient confusion to help formally analyze this. When gradient confusion is high, stochastic gradients produced by different data samples may be negatively correlated, slowing down convergence. But when gradient confusion is low, data samples interact harmoniously, and training proceeds quickly. Through theoretical and experimental results, we demonstrate how the neural network architecture affects gradient confusion, and thus the efficiency of training. Our results show that, for popular initialization techniques, increasing the width of neural networks leads to lower gradient confusion, and thus faster model training. On the other hand, increasing the depth of neural networks has the opposite effect. Our results indicate that alternate initialization techniques or networks using both batch normalization and skip connections help reduce the training burden of very deep networks.

Motivation & Objective

• Motivate and formalize the notion of gradient confusion as a factor in SGD convergence on over-parameterized networks.
• Analyze how architecture choices (width, depth) influence gradient confusion under Gaussian initialization.
• Show theoretical bounds linking gradient confusion to SGD convergence rates and training speed.
• Provide empirical validation across WRNs, CNNs, and MLPs on CIFAR/MNIST to connect theory with practice.

Proposed method

• Define gradient confusion as a bound on pairwise gradient inner products across mini-batches.
• Establish convergence results for constant learning rate SGD with gradient confusion bound under PL inequality and Lipschitz smoothness.
• Prove that gradient confusion grows with depth and decreases with width under Gaussian initialization.
• Extend results to general settings with small weight assumption and uniform sphere data sampling.
• Demonstrate that orthogonal initialization can make deep linear nets' gradient confusion depth-independent.
• Perform extensive experiments measuring gradient cosine similarities and training convergence on WRNs, CNNs, and MLPs to validate theory.

Experimental results

Research questions

• RQ1 How does gradient confusion quantify the interaction of mini-batch gradients during SGD on over-parameterized networks?
• RQ2 How do width and depth affect gradient confusion under standard Gaussian initializations?
• RQ3 Can architectural changes like batch normalization and skip connections reduce gradient confusion and improve trainability?
• RQ4 Do results extend to linear networks with orthogonal initialization and to non-initial training regimes?
• RQ5 What empirical patterns in gradient similarity emerge in common architectures on benchmark datasets?

Key findings

• Gradient confusion connects architecture to SGD speed: higher confusion slows convergence, while lower confusion accelerates it.
• Under Gaussian initializations, increasing network depth raises gradient confusion, while increasing width lowers it.
• Batch normalization and skip connections together significantly lower gradient confusion in very deep nets, improving trainability.
• For deep linear networks with orthogonal initialization, gradient confusion is independent of depth.
• Experimental results on WRNs, CNNs, and MLPs show wider networks train faster and gradient similarities concentrate near zero with width.
• The theory explains why architectures with residual connections and normalization enable efficient training with constant learning rates.