[Paper Review] SGDR: Stochastic Gradient Descent with Warm Restarts

Ilya Loshchilov, Frank Hutter|arXiv (Cornell University)|Aug 13, 2016

Domain Adaptation and Few-Shot Learning28 references1,741 citations

TL;DR

Introduces stochastic gradient descent with cosine-annealed warm restarts to improve training speed and generalization for deep networks, achieving state-of-the-art results on CIFAR-10/100 and enabling effective snapshot ensembles.

ABSTRACT

Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at https://github.com/loshchil/SGDR

Motivation & Objective

• Motivate and address the slow anytime performance of SGD in deep neural network training.
• Propose a simple warm restart mechanism with cosine annealing to improve convergence speed.
• Demonstrate improvements on CIFAR-10/100 and show benefits on EEG data and a downsampled ImageNet.
• Explore ensemble gains from snapshots taken during SGDR trajectories.
• Highlight potential for enabling faster architecture exploration and training efficiency.

Proposed method

• Simulate warm restarts in SGD by restarting with an increased learning rate at predefined intervals without resetting model weights.
• Within each restart, apply cosine annealing to the learning rate from a max to a min value over T_i epochs: eta_t = eta_min^i + 0.5*(eta_max^i - eta_min^i)*(1 + cos(T_cur/T_i * pi)).
• Allow T_i to grow via a multiplier T_mult to improve anytime performance and enable faster attainment of good test error.
• Use a single or few SGDR runs with identical eta_max/min across restarts to reduce hyperparameter tuning.
• Optionally ensemble models from SGDR snapshots taken before restarts to form ensembles with improved accuracy.
• Compare SGDR against standard learning rate schedules and reproduce baseline results on WRN architectures.

Experimental results

Research questions

• RQ1 Does SGDR improve the training efficiency (time to reach a target test error) compared to standard SGD schedules?
• RQ2 Can cosine-annealed warm restarts and restarts with increasing T_i yield faster convergence and better generalization?
• RQ3 Do snapshot ensembles from SGDR trajectories provide substantial gains over single-run models or ensembles from independent runs?
• RQ4 Are SGDR gains transferable to domains beyond CIFAR (e.g., EEG data) and to downsampled ImageNet configurations?
• RQ5 What are the practical hyperparameters (initial learning rate, T_i, T_mult) that balance speed and accuracy?

Key findings

• SGDR achieves faster attainment of competitive test errors on CIFAR-10 (~4% area) and CIFAR-100 (~20%) compared to the default schedule.
• Ensembles formed from SGDR snapshots yield state-of-the-art-like improvements (e.g., 3.14% test error on CIFAR-10 and 16.21% on CIFAR-100 with N=16 runs and M=3 snapshots).
• SGDR enables training wider networks (WRN-28-20) to achieve better accuracy within similar or shorter budgets than narrower ones trained with standard schedules.
• Snapshots from SGDR provide diverse, useful ensemble members that outperform equivalent ensembles built from independent runs in many settings.
• Preliminary experiments show SGDR improves performance on EEG datasets and a downsampled ImageNet, indicating broader applicability.