[Paper Review] On the Noisy Gradient Descent that Generalizes as SGD

Jingfeng Wu, Wenqing Hu+4 more|arXiv (Cornell University)|Jun 18, 2019

Sparse and Compressive Sensing Techniques52 references19 citations

TL;DR

This paper proposes a novel framework for noisy gradient descent by modeling gradient noise as the product of the gradient matrix and a sampling noise, demonstrating that the noise class is not critical for generalization as long as magnitude and covariance structure are appropriately set. The key contribution is a computationally efficient algorithm, [MSGD-Fisher]-B, that closes the generalization gap in large-batch training by injecting sparse Gaussian sampling noise, outperforming standard large-batch SGD and matching small-batch SGD performance.

ABSTRACT

The gradient noise of SGD is considered to play a central role in the observed strong generalization abilities of deep learning. While past studies confirm that the magnitude and the covariance structure of gradient noise are critical for regularization, it remains unclear whether or not the class of noise distributions is important. In this work we provide negative results by showing that noises in classes different from the SGD noise can also effectively regularize gradient descent. Our finding is based on a novel observation on the structure of the SGD noise: it is the multiplication of the gradient matrix and a sampling noise that arises from the mini-batch sampling procedure. Moreover, the sampling noises unify two kinds of gradient regularizing noises that belong to the Gaussian class: the one using (scaled) Fisher as covariance and the one using the gradient covariance of SGD as covariance. Finally, thanks to the flexibility of choosing noise class, an algorithm is proposed to perform noisy gradient descent that generalizes well, the variant of which even benefits large batch SGD training without hurting generalization.

Motivation & Objective

• To investigate whether the class of noise distribution is essential for the generalization performance of stochastic gradient descent (SGD).
• To understand the origin of gradient noise in SGD by decomposing it into the gradient matrix and a sampling noise component.
• To develop a practical, efficient algorithm for injecting structured noise that enables large-batch SGD to generalize as well as small-batch SGD.
• To unify existing Gaussian noise-based regularization methods, such as those using scaled Fisher or gradient covariance, under a common sampling noise framework.

Proposed method

• The paper introduces a decomposition of SGD gradient noise as the product of the gradient matrix and a sampling noise arising from mini-batch selection.
• It identifies that the sampling noise, not the gradient matrix alone, determines the effective noise distribution in SGD.
• The authors propose [MSGD-Fisher]-B, a variant of mini-batch MSGD that injects sparse Gaussian sampling noise to compensate for the lack of noise in large-batch training.
• The method avoids costly SVD-based noise injection by using a structured sampling noise that preserves the covariance structure of the gradient matrix.
• Theoretical analysis shows that strong convergence between discrete iterations and continuous SDEs is achievable with Gaussian MSGD, enabling faster convergence.
• Empirical validation is conducted on both toy datasets and real-world models like ResNet-18 on CIFAR-10 to compare generalization and convergence speed.

Experimental results

Research questions

• RQ1 Is the class of noise distribution a critical factor in the generalization performance of stochastic optimization methods?
• RQ2 How does the sampling procedure in mini-batch SGD give rise to the observed gradient noise structure?
• RQ3 Can alternative noise classes—beyond Gaussian—effectively regularize gradient descent when magnitude and covariance are properly controlled?
• RQ4 Can a computationally efficient algorithm be designed to inject compensatory noise that restores generalization in large-batch SGD?
• RQ5 How do different noise distributions compare in terms of convergence speed and final generalization performance?

Key findings

• The noise class is not a decisive factor for regularization; instead, noise magnitude and covariance structure are more critical for generalization.
• The gradient noise in SGD arises from the product of the gradient matrix and a sampling noise, which unifies two existing Gaussian noise approaches: one using scaled Fisher and the other using gradient covariance as covariance.
• The proposed [MSGD-Fisher]-B algorithm successfully closes the generalization gap in large-batch training, achieving performance comparable to small-batch SGD on CIFAR-10 with ResNet-18.
• Gaussian MSGD converges faster than other noise types, likely due to strong convergence between discrete iterations and continuous SDEs.
• The gradient matrix preserves essential structural information for regularization, as evidenced by the superior performance of MSGD-Bernoulli/Fisher compared to GLD-diag, which destroys this structure.
• Experiments show that even with tuned noise magnitude, large-batch SGD without compensation generalizes poorly, while [MSGD-Fisher]-B restores generalization without sacrificing training speed.