[Paper Review] On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

Nitish Shirish Keskar, Dheevatsa Mudigere+3 more|arXiv (Cornell University)|Sep 15, 2016

Stochastic Gradient Optimization Techniques34 references577 citations

TL;DR

The paper shows that large-batch SGD tends to converge to sharp minima, causing generalization gaps, while small-batch methods find flatter minima; gradient noise helps large-batch methods explore and potentially close the gap.

ABSTRACT

The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.

Motivation & Objective

• Motivate and quantify the generalization gap observed when using large mini-batches in SGD for deep learning.
• Investigate whether large-batch methods converge to sharp minima and how this relates to poorer generalization.
• Compare the minimizers found by small-batch versus large-batch training across multiple network architectures.
• Offer potential remedies and practical insights to improve large-batch training without sacrificing generalization.

Proposed method

• Define SB and LB training regimes and use ADAM to compare their behavior across six network/data-set configurations.
• Use a sharpness/sensitivity metric based on perturbations in local neighborhoods to characterize minimizers.
• Produce parametric plots along lines between SB and LB solutions to illustrate minimizer sharpness.
• Conduct warm-start experiments to test how SB exploration can influence LB outcomes.
• Analyze batch-size thresholds and their effect on generalization and sharpness.

Experimental results

Research questions

• RQ1 Does large-batch training lead to sharp minimizers that degrade generalization?
• RQ2 How do SB and LB minimizers differ in terms of sharpness and local landscape structure?
• RQ3 Can gradient noise from SB training help LB methods escape sharp basins and improve generalization?
• RQ4 What practical strategies might mitigate the generalization drop associated with LB training?

Key findings

• LB methods converge to sharp minimizers characterized by large positive Hessian eigenvalues and reduced generalization.
• SB methods converge to flatter minimizers with many small eigenvalues and better generalization.
• Parametric and subspace-sharpness analyses show LB minimizers are markedly sharper than SB minimizers on multiple networks.
• Warm-start experiments indicate SB exploration can enable LB to reach flat minimizers if LB is started after sufficient SB exploration.
• There exists a threshold batch size beyond which LB performance deteriorates in testing accuracy for several networks.
• Remedies like data augmentation and adversarial training improve LB generalization somewhat but do not fully eliminate sharp minima.