[논문 리뷰] The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent

Karthik Abinav Sankararaman, Soham De외 3명|arXiv (Cornell University)|2019. 04. 15.

Stochastic Gradient Optimization Techniques참고 문헌 72인용 수 35

한 줄 요약

본 논문은 과잉 매개화 네트워크에서 SGD 다이나믹스를 분석하기 위한 기울기 혼동(gradient confusion) 개념을 정의하고, 너비가 깊이를 감소시키는 반면 깊이가 혼동을 증가시킴을 보인다. 배치 정규화 및 건너뛰기 연결과 같은 기법이 깊이에 따른 학습 부담을 완화한다.

ABSTRACT

This paper studies how neural network architecture affects the speed of training. We introduce a simple concept called gradient confusion to help formally analyze this. When gradient confusion is high, stochastic gradients produced by different data samples may be negatively correlated, slowing down convergence. But when gradient confusion is low, data samples interact harmoniously, and training proceeds quickly. Through theoretical and experimental results, we demonstrate how the neural network architecture affects gradient confusion, and thus the efficiency of training. Our results show that, for popular initialization techniques, increasing the width of neural networks leads to lower gradient confusion, and thus faster model training. On the other hand, increasing the depth of neural networks has the opposite effect. Our results indicate that alternate initialization techniques or networks using both batch normalization and skip connections help reduce the training burden of very deep networks.

연구 동기 및 목표

• Motivate and formalize the notion of gradient confusion as a factor in SGD convergence on over-parameterized networks.
• Analyze how architecture choices (width, depth) influence gradient confusion under Gaussian initialization.
• Show theoretical bounds linking gradient confusion to SGD convergence rates and training speed.
• Provide empirical validation across WRNs, CNNs, and MLPs on CIFAR/MNIST to connect theory with practice.

제안 방법

• Define gradient confusion as a bound on pairwise gradient inner products across mini-batches.
• Establish convergence results for constant learning rate SGD with gradient confusion bound under PL inequality and Lipschitz smoothness.
• Prove that gradient confusion grows with depth and decreases with width under Gaussian initialization.
• Extend results to general settings with small weight assumption and uniform sphere data sampling.
• Demonstrate that orthogonal initialization can make deep linear nets' gradient confusion depth-independent.
• Perform extensive experiments measuring gradient cosine similarities and training convergence on WRNs, CNNs, and MLPs to validate theory.

실험 결과

연구 질문

• RQ1 How does gradient confusion quantify the interaction of mini-batch gradients during SGD on over-parameterized networks?
• RQ2 How do width and depth affect gradient confusion under standard Gaussian initializations?
• RQ3 Can architectural changes like batch normalization and skip connections reduce gradient confusion and improve trainability?
• RQ4 Do results extend to linear networks with orthogonal initialization and to non-initial training regimes?
• RQ5 What empirical patterns in gradient similarity emerge in common architectures on benchmark datasets?

주요 결과

• Gradient confusion connects architecture to SGD speed: higher confusion slows convergence, while lower confusion accelerates it.
• Under Gaussian initializations, increasing network depth raises gradient confusion, while increasing width lowers it.
• Batch normalization and skip connections together significantly lower gradient confusion in very deep nets, improving trainability.
• For deep linear networks with orthogonal initialization, gradient confusion is independent of depth.
• Experimental results on WRNs, CNNs, and MLPs show wider networks train faster and gradient similarities concentrate near zero with width.
• The theory explains why architectures with residual connections and normalization enable efficient training with constant learning rates.