[Paper Review] Weight-averaged consistency targets improve semi-supervised deep learning results.

Antti Tarvainen, Harri Valpola|arXiv (Cornell University)|Mar 6, 2017

Advanced Neural Network ApplicationsComputer Science62 citations

TL;DR

This paper proposes Mean Teacher, a semi-supervised learning method that improves generalization by averaging model weights across training iterations to create consistent target predictions. By updating these targets more frequently than Temporal Ensembling, it achieves state-of-the-art results: 4.35% error on SVHN with only 250 labels and 6.28% on CIFAR-10 with 4,000 labels, outperforming prior methods.

ABSTRACT

The recently proposed Temporal Ensembling has achieved state-of-the-art results in several semi-supervised learning benchmarks. It maintains an exponential moving average of label predictions on each training example, and penalizes predictions that are inconsistent with this target. However, because the targets change only once per epoch, Temporal Ensembling becomes unwieldy when learning large datasets. To overcome this problem, we propose Mean Teacher, a method that averages model weights instead of label predictions. As an additional benefit, Mean Teacher improves test accuracy and enables training with fewer labels than Temporal Ensembling. Without changing the network architecture, Mean Teacher achieves an error rate of 4.35% on SVHN with 250 labels, outperforming Temporal Ensembling trained with 1000 labels. We also show that a good network architecture is crucial to performance. Combining Mean Teacher and Residual Networks, we improve the state of the art on CIFAR-10 with 4000 labels from 10.55% to 6.28%, and on ImageNet 2012 with 10% of the labels from 35.24% to 9.11%.

Motivation & Objective

• To address the inefficiency of Temporal Ensembling in large-scale datasets due to infrequent target updates.
• To improve semi-supervised learning performance by replacing prediction-based consistency with weight-averaged model consistency.
• To reduce the number of labeled examples required for high accuracy in semi-supervised training.
• To demonstrate that combining Mean Teacher with strong architectures like Residual Networks further enhances performance.

Proposed method

• Mean Teacher maintains an exponential moving average of the model's own weights to form a teacher network.
• During training, the student network's predictions are regularized to match the teacher network's predictions on the same input.
• The teacher network's weights are updated using a momentum update rule: θ_teacher ← τθ_teacher + (1−τ)θ_student.
• The method applies consistency regularization by minimizing the L2 loss between student and teacher predictions on the same augmented input.
• The approach enables frequent target updates, improving training stability and convergence on large datasets.
• The method is architecture-agnostic and can be combined with any deep neural network, especially residual networks.

Experimental results

Research questions

• RQ1 Can weight-averaged consistency targets improve semi-supervised learning performance compared to prediction-averaged targets?
• RQ2 Does frequent target update via weight averaging lead to better generalization and faster convergence?
• RQ3 Can Mean Teacher achieve state-of-the-art results with significantly fewer labeled examples than prior methods?
• RQ4 How does the combination of Mean Teacher and residual networks affect performance on standard benchmarks?

Key findings

• Mean Teacher achieves a test error of 4.35% on SVHN using only 250 labeled examples, surpassing Temporal Ensembling trained with 1,000 labels.
• On CIFAR-10 with 4,000 labels, Mean Teacher reduces error from 10.55% to 6.28% when combined with residual networks.
• On ImageNet 2012 with 10% of the training labels, the error drops from 35.24% to 9.11% using Mean Teacher and residual networks.
• The method enables more frequent target updates than Temporal Ensembling, improving training efficiency on large-scale datasets.
• The performance gain is attributed not only to the consistency mechanism but also to the use of a strong residual network architecture.