[Paper Review] Snapshot Ensembles: Train 1, get M for free

Gao Huang, Yixuan Li+4 more|arXiv (Cornell University)|Apr 1, 2017

Advanced Neural Network Applications118 citations

TL;DR

This paper trains one model with cyclic cosine learning rate schedules to obtain multiple well-performing snapshots, which are ensembled at test time to improve accuracy without extra training cost.

ABSTRACT

Ensembles of neural networks are known to be much more robust and accurate than individual networks. However, training multiple deep networks for model averaging is computationally expensive. In this paper, we propose a method to obtain the seemingly contradictory goal of ensembling multiple neural networks at no additional training cost. We achieve this goal by training a single neural network, converging to several local minima along its optimization path and saving the model parameters. To obtain repeated rapid convergence, we leverage recent work on cyclic learning rate schedules. The resulting technique, which we refer to as Snapshot Ensembling, is simple, yet surprisingly effective. We show in a series of experiments that our approach is compatible with diverse network architectures and learning tasks. It consistently yields lower error rates than state-of-the-art single models at no additional training cost, and compares favorably with traditional network ensembles. On CIFAR-10 and CIFAR-100 our DenseNet Snapshot Ensembles obtain error rates of 3.4% and 17.4% respectively.

Motivation & Objective

• Motivate reducing training cost for high-accuracy ensembles.
• Leverage SGD’s visits to multiple local minima to create explicit snapshots.
• Use cyclic cosine learning rate schedules to generate and save multiple model snapshots.
• Demonstrate that ensembling snapshots improves performance across architectures and datasets without extra training time.
• Show compatibility with data augmentation, stochastic depth, and batch normalization.

Proposed method

• Train a single network over M learning-rate cycles using a cosine annealing schedule.
• Take and save a snapshot of the model parameters at the end of each cycle.
• At test time, average the last m snapshots to form the ensemble.
• Cycle length and restart rate controlled by a cyclic learning-rate function per Loshchilov & Hutter (2016).
• Keep total training time equal to a standard single-model training; gain ensemble benefits at no extra training cost.
• Evaluate on ResNet, Wide-ResNet, and DenseNet architectures across CIFAR-10, CIFAR-100, SVHN, Tiny ImageNet, and ImageNet.

Experimental results

Research questions

• RQ1 Can an ensemble be formed from multiple local minima found along a single optimization path without increasing training cost?
• RQ2 Does a cyclic cosine learning-rate schedule yield diverse, complementary snapshots suitable for ensembling?
• RQ3 What is the impact of the number of cycles M and restart scale on ensemble performance across datasets and architectures?
• RQ4 How does Snapshot Ensemble compare to traditional ensembles and implicit ensembling methods under the same training budget?

Key findings

• Snapshot Ensembles consistently reduce error rates compared with baselines trained for the same budget.
• On CIFAR-10 and CIFAR-100 with DenseNet, they report 3.44% and 17.41% error respectively for certain settings.
• On ImageNet (ResNet-50), a two-cycle Snapshot Ensemble achieves 23.33% vs 24.01% for a single model.
• Ensembles using 2–8 cycles can outperform single models, with diminishing returns beyond very large ensembles.
• NoCycle variants underperform, highlighting the importance of cyclic learning-rate schedules for effective diversification.
• The method yields accuracy gains with minimal or zero extra training cost and is compatible with other accuracy-improving techniques.