[Paper Review] Interplay Between Optimization and Generalization of Stochastic Gradient Descent with Covariance Noise.

Yeming Wen, Kevin Luk+4 more|arXiv (Cornell University)|Feb 21, 2019

Stochastic Gradient Optimization Techniques32 references19 citations

TL;DR

This paper proposes adding covariance noise to stochastic gradient descent (SGD) gradients to improve generalization in large-batch training without sacrificing optimization speed. It shows that the noise's structure—specifically its Frobenius norm—better captures performance than gradient variance, and empirically validates that this method enhances generalization while maintaining fast convergence.

ABSTRACT

The choice of batch-size in a stochastic optimization algorithm plays a substantial role for both optimization and generalization. Increasing the batch-size used typically improves optimization but degrades generalization. To address the problem of improving generalization while maintaining optimal convergence in large-batch training, we propose to add covariance noise to the gradients. We demonstrate that the learning performance of our method is more accurately captured by the structure of the covariance matrix of the noise rather than by the variance of gradients. Moreover, over the convex-quadratic, we prove in theory that it can be characterized by the Frobenius norm of the noise matrix. Our empirical studies with standard deep learning model-architectures and datasets shows that our method not only improves generalization performance in large-batch training, but furthermore, does so in a way where the optimization performance remains desirable and the training duration is not elongated.

Motivation & Objective

• To address the trade-off between optimization efficiency and generalization in large-batch stochastic gradient descent (SGD).
• To investigate whether structured noise—specifically covariance noise—can improve generalization without degrading optimization speed.
• To determine whether the performance of noisy SGD is better characterized by the noise's covariance structure rather than its gradient variance.
• To theoretically and empirically validate that the Frobenius norm of the noise matrix captures learning performance in convex-quadratic settings.
• To demonstrate that the proposed method maintains fast convergence while improving test accuracy in standard deep learning models.

Proposed method

• Introduces covariance noise by adding a noise matrix to the gradient updates in SGD, where the noise matrix is drawn from a distribution with specified covariance structure.
• The noise matrix is designed such that its covariance structure directly influences the update direction and variance in parameter space.
• Theoretical analysis focuses on convex-quadratic problems, proving that generalization performance is characterized by the Frobenius norm of the noise matrix.
• Empirical evaluation uses standard deep learning architectures (e.g., ResNet, VGG) on benchmark datasets (e.g., CIFAR-10, ImageNet) with varying batch sizes.
• Noise is added during training with controlled covariance, and performance is measured in terms of test accuracy and convergence speed.
• The method is compared against standard SGD and other noise injection techniques, with ablation studies on noise variance and structure.

Experimental results

Research questions

• RQ1 Can adding covariance noise to SGD gradients improve generalization in large-batch training without slowing convergence?
• RQ2 Is the performance of noisy SGD better predicted by the noise's covariance structure or its gradient variance?
• RQ3 In convex-quadratic problems, does the Frobenius norm of the noise matrix characterize learning performance?
• RQ4 Does the proposed method maintain fast optimization dynamics while improving test accuracy in standard deep learning setups?
• RQ5 How does the structure of the noise matrix compare to isotropic noise in terms of generalization gains?

Key findings

• The addition of covariance noise significantly improves generalization in large-batch SGD, enabling performance comparable to small-batch training.
• Generalization performance is better captured by the Frobenius norm of the noise matrix than by the variance of the gradients in convex-quadratic problems.
• Theoretical analysis confirms that the Frobenius norm of the noise matrix characterizes learning performance in convex-quadratic settings.
• Empirical results show that the method maintains fast convergence and does not increase training duration, even with large batch sizes.
• The method achieves higher test accuracy than standard SGD and other noise injection baselines across multiple architectures and datasets.
• The structure of the noise matrix—specifically its covariance—proves more effective than isotropic noise for improving generalization.