[Paper Review] Don't Decay the Learning Rate, Increase the Batch Size

Samuel Smith, Pieter-Jan Kindermans+2 more|arXiv (Cornell University)|Nov 1, 2017

Advanced Neural Network ApplicationsComputer Science23 references391 citations

TL;DR

The paper shows that increasing the batch size during training at a fixed learning rate can replicate the learning curve and test accuracy of decaying learning rate schedules, enabling large-batch training with fewer updates.

ABSTRACT

It is common practice to decay the learning rate. Here we show one can usually obtain the same learning curve on both training and test sets by instead increasing the batch size during training. This procedure is successful for stochastic gradient descent (SGD), SGD with momentum, Nesterov momentum, and Adam. It reaches equivalent test accuracies after the same number of training epochs, but with fewer parameter updates, leading to greater parallelism and shorter training times. We can further reduce the number of parameter updates by increasing the learning rate $ε$ and scaling the batch size $B \propto ε$. Finally, one can increase the momentum coefficient $m$ and scale $B \propto 1/(1-m)$, although this tends to slightly reduce the test accuracy. Crucially, our techniques allow us to repurpose existing training schedules for large batch training with no hyper-parameter tuning. We train ResNet-50 on ImageNet to $76.1\%$ validation accuracy in under 30 minutes.

Motivation & Objective

• Motivate and understand why decaying the learning rate is commonly used in SGD-based optimization.
• Propose and validate an alternative: increase batch size during training to match learning curves and generalization.
• Demonstrate that large batch training can achieve similar test accuracy with fewer parameter updates across optimizers.
• Show practical scalability benefits on CIFAR-10 and ImageNet with different architectures and hardware.

Proposed method

• Model SGD as a stochastic differential equation to analyze noise scale g = ε(N/B − 1).
• Demonstrate equivalence between decaying learning rate and increasing batch size for fixed training epochs.
• Propose and test schedules where batch size increases as learning rate would decay (B ∝ α when ε decays by α).
• Explore effective learning rate ε_eff = ε/(1−m) for momentum and adjust batch size accordingly.
• Empirically validate on CIFAR-10 with Wide ResNet and on ImageNet with Inception-ResNet-V2 and ResNet-50/TPU.
• Assess impact of momentum on large-batch training and discuss accumulation dynamics.

Experimental results

Research questions

• RQ1 Can increasing the batch size during training at a fixed learning rate replicate the effects of decaying learning rate schedules on both training dynamics and test accuracy?
• RQ2 How does adjusting batch size, learning rate, and momentum interact to minimize parameter updates while preserving generalization?
• RQ3 What are the practical limits and benefits of large-batch training across common optimizers (SGD, momentum, Nesterov, Adam) and architectures?
• RQ4 How do these strategies translate to ImageNet-scale training on GPUs and TPUs?

Key findings

• Increasing batch size during training yields near-identical test accuracies to decaying learning rate schedules for SGD, SGD with momentum, Nesterov momentum, and Adam.
• Replacing learning-rate decay with batch-size growth reduces the number of parameter updates while maintaining performance.
• Using larger learning rate and scaling batch size as B ∝ ε or B ∝ 1/(1−m) can further reduce updates, with some trade-offs in accuracy depending on momentum.
• On CIFAR-10, wide ResNet experiments show identical training curves when comparing decayed LR, hybrid, and increasing-B schedules.
• On ImageNet, large-batch training with 65,536 per batch achieved 77–77.5% validation accuracy with under 2,500 updates (Inception-ResNet-V2) and 76.1% on ResNet-50 in under 30 minutes on TPU.
• Large-batch training can drastically reduce wall-clock time without extra hyperparameter tuning.