[Paper Review] On Minibatch Noise: Discrete-Time SGD, Overparametrization, and Bayes.

Liu Ziyin, Kangqiao Liu+2 more|arXiv (Cornell University)|Feb 10, 2021

Machine Learning and Data Classification4 citations

TL;DR

This paper investigates the conditions under which minibatch noise emerges in stochastic gradient descent (SGD), showing that a mismatch between model and data complexity—due to label noise, input noise, regularization, or underparametrization—is necessary for noise to appear. It derives analytically solvable results for linear regression and proposes a more accurate general formulation for minibatch noise beyond common approximations.

ABSTRACT

The noise in stochastic gradient descent (SGD), caused by minibatch sampling, remains poorly understood despite its enormous practical importance in offering good training efficiency and generalization ability. In this work, we study the minibatch noise in SGD. Motivated by the observation that minibatch sampling does not always a fluctuation, we set out to find the conditions that minibatch noise to emerge. We first derive the analytically solvable results for linear regression under various settings, which are compared to the commonly used approximations that are used to understand SGD noise. We show that some degree of mismatch between model and data complexity is needed in order for SGD to cause a noise, and that such mismatch may be due to the existence of static noise in the labels, in the input, the use of regularization, or underparametrization. Our results motivate a more accurate general formulation to describe minibatch noise.

Motivation & Objective

• To understand why minibatch noise in SGD is not always present, despite its central role in training efficiency and generalization.
• To identify the specific conditions under which minibatch noise actually emerges during optimization.
• To derive analytically tractable results for minibatch noise in linear regression under various settings.
• To challenge and improve upon commonly used approximations of SGD noise by proposing a more accurate general formulation.

Proposed method

• Analytically derive the noise covariance in SGD for linear regression under different settings, including with and without regularization.
• Compare exact analytical results against standard approximations used in the literature, such as the constant Fisher information approximation.
• Analyze the role of model complexity relative to data complexity, particularly in the presence of label or input noise.
• Investigate how regularization and underparametrization contribute to the emergence of minibatch noise.
• Formulate a general noise model that accounts for mismatched model-data complexity, extending beyond simplified assumptions.

Experimental results

Research questions

• RQ1 Under what conditions does minibatch noise actually emerge in SGD, rather than averaging out?
• RQ2 How does the mismatch between model and data complexity influence the emergence of minibatch noise?
• RQ3 To what extent do common approximations of SGD noise fail to capture the true noise behavior in linear models?
• RQ4 What role do label noise, input noise, regularization, and underparametrization play in enabling minibatch noise?
• RQ5 Can a more accurate general formulation of minibatch noise be derived that accounts for model-data complexity mismatch?

Key findings

• Minibatch noise in SGD does not emerge unless there is a mismatch between model and data complexity, such as in the presence of label or input noise.
• A degree of underparametrization or regularization is necessary for noise to appear, as overparametrized models without noise components may not exhibit fluctuating gradients.
• Common approximations of SGD noise, such as assuming constant Fisher information, fail to capture the true noise dynamics when model-data mismatch is present.
• Analytical results for linear regression show that noise variance depends explicitly on the level of model-data complexity mismatch, including regularization strength and noise variance in inputs or labels.
• The study motivates a revised general formulation of minibatch noise that incorporates model complexity relative to data, improving accuracy over prior approximations.