[Paper Review] Temporal Ensembling for Semi-Supervised Learning

Samuli Laine, Timo Aila|arXiv (Cornell University)|Oct 7, 2016

Advanced Neural Network Applications30 references1,496 citations

TL;DR

The paper introduces self-ensembling methods (Pi-model and temporal ensembling) that use ensemble predictions across epochs and augmentations as targets for unlabeled data, achieving state-of-the-art semi-supervised results.

ABSTRACT

In this paper, we present a simple and efficient method for training deep neural networks in a semi-supervised setting where only a small portion of training data is labeled. We introduce self-ensembling, where we form a consensus prediction of the unknown labels using the outputs of the network-in-training on different epochs, and most importantly, under different regularization and input augmentation conditions. This ensemble prediction can be expected to be a better predictor for the unknown labels than the output of the network at the most recent training epoch, and can thus be used as a target for training. Using our method, we set new records for two standard semi-supervised learning benchmarks, reducing the (non-augmented) classification error rate from 18.44% to 7.05% in SVHN with 500 labels and from 18.63% to 16.55% in CIFAR-10 with 4000 labels, and further to 5.12% and 12.16% by enabling the standard augmentations. We additionally obtain a clear improvement in CIFAR-100 classification accuracy by using random images from the Tiny Images dataset as unlabeled extra inputs during training. Finally, we demonstrate good tolerance to incorrect labels.

Motivation & Objective

• Motivate semi-supervised learning where only a small portion of data is labeled.
• Propose self-ensembling approaches that leverage dropout and input augmentation to generate reliable targets for unlabeled data.
• Demonstrate improved classification accuracy on standard benchmarks (SVHN, CIFAR-10) over prior methods.
• Show tolerance to label noise and potential gains in fully supervised settings.
• Discuss practical considerations and connections to related semi-supervised architectures.

Proposed method

• Two self-ensembling implementations are proposed: Pi-model and temporal ensembling.
• Pi-model enforces consistency between two outputs of the same input under different dropout and augmentations.
• Temporal ensembling aggregates predictions across multiple previous epochs to form stable targets for unlabeled data.
• The unsupervised loss compares current predictions to ensemble targets using mean-squared error on the full output vector, scaled by a ramped weight w(t).
• Dropout-based regularization and input augmentation are essential to obtain meaningful ensemble targets.
• Temporal ensembling updates ensemble predictions Z with momentum and applies bias correction to form targets tilde{z}.

Experimental results

Research questions

• RQ1 Can self-ensembling via multiple epochs and augmentations improve semi-supervised learning when labels are scarce?
• RQ2 How do Pi-model and temporal ensembling compare in terms of accuracy and training efficiency?
• RQ3 What is the role of dropout, augmentation, and ramp-up of the unsupervised loss in avoiding degenerate solutions?
• RQ4 Do these methods extend to fully supervised settings or tolerate noisy labels?

Key findings

• The Pi-model and temporal ensembling surpass prior semi-supervised methods on SVHN and CIFAR-10 benchmarks.
• With 500 labels on SVHN, error drops from 18.44% to 7.05% in non-augmented setup.
• With 4000 labels on CIFAR-10, error drops from 18.63% to 16.55% in non-augmented setup.
• Enabling standard augmentations further reduces errors to 5.12% (SVHN) and 12.16% (CIFAR-10).
• Temporal ensembling provides faster training and can yield better results than Pi-model given similar epochs.
• The methods show tolerance to incorrect labels and relate to existing semi-supervised frameworks like ladder networks and bootstrapping.