[Paper Review] A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay

Leslie N. Smith|arXiv (Cornell University)|Mar 26, 2018

Advanced Neural Network ApplicationsComputer Science18 references822 citations

TL;DR

The paper presents practical, efficient methods to set learning rate, batch size, momentum, and weight decay by analyzing validation/test loss, using cyclical learning rates/momentum, and balancing regularization to accelerate training while improving performance.

ABSTRACT

Although deep learning has produced dazzling successes for applications of image, speech, and video processing in the past few years, most trainings are with suboptimal hyper-parameters, requiring unnecessarily long training times. Setting the hyper-parameters remains a black art that requires years of experience to acquire. This report proposes several efficient ways to set the hyper-parameters that significantly reduce training time and improves performance. Specifically, this report shows how to examine the training validation/test loss function for subtle clues of underfitting and overfitting and suggests guidelines for moving toward the optimal balance point. Then it discusses how to increase/decrease the learning rate/momentum to speed up training. Our experiments show that it is crucial to balance every manner of regularization for each dataset and architecture. Weight decay is used as a sample regularizer to show how its optimal value is tightly coupled with the learning rates and momentums. Files to help replicate the results reported here are available.

Motivation & Objective

• Reduce training time and improve performance by disciplined hyper-parameter tuning.
• Use training validation/test loss to diagnose underfitting/overfitting early in training.
• Show interdependence of learning rate, momentum, batch size, and weight decay and how to balance them.
• Introduce cyclical learning rates (CLR) and cyclical momentum (CM) and the 1cycle policy to accelerate convergence.
• Provide practical guidelines and replication resources for practitioners.

Proposed method

• Analyze training and validation/test loss early in training to guide hyper-parameter adjustments.
• Employ cyclical learning rate (CLR) range tests to identify suitable learning rate bounds.
• Use the 1cycle learning rate policy to enable fast convergence with large learning rates.
• Investigate cyclical momentum (CM) and its interaction with CLR to stabilize training.
• Evaluate weight decay and its balance with LR and CM across datasets and architectures.
• Offer replication files and practical guidelines for practitioners.

Experimental results

Research questions

• RQ1 How can practitioners efficiently determine optimal learning rate, batch size, momentum, and weight decay without exhaustive grid searches?
• RQ2 What early indicators in validation/test loss reveal underfitting or overfitting during training?
• RQ3 How do cyclical learning rates and cyclical momentum interact to affect convergence speed and stability?
• RQ4 What is the role of weight decay in balancing regularization with other hyper-parameters across architectures and datasets?

Key findings

• Validation/test loss provides information about convergence and generalization not always visible in training loss or accuracy.
• LR range tests help identify the maximum usable learning rate and optimal LR bounds for CLR.
• Large learning rates can enable faster training (super-convergence) when other regularizations are reduced to balance regularization.
• Batch size interacts with LR and hardware constraints; larger batch sizes can improve final accuracy given near-constant execution time, with diminishing returns beyond a point.
• Cyclical momentum combined with CLR often yields better robustness and final performance than constant momentum, especially for deeper networks like ResNet-56.
• Weight decay should be balanced with learning rate and momentum; optimal value is dataset- and architecture-dependent and benefits from joint exploration with CLR/CM.