[Paper Review] There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average

Ben Athiwaratkun, Marc Finzi+2 more|arXiv (Cornell University)|Jun 14, 2018

Machine Learning and Data Classification149 citations

TL;DR

The paper analyzes consistency-based semi-supervised learning and shows that SGD explores many solutions; averaging weights via SWA and fast-SWA improves generalization, achieving state-of-the-art results on CIFAR-10/100 with limited labels.

ABSTRACT

Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters. To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedures. The consistency loss dramatically improves generalization performance over supervised-only training; however, we show that SGD struggles to converge on the consistency loss and continues to make large steps that lead to changes in predictions on the test data. Motivated by these observations, we propose to train consistency-based methods with Stochastic Weight Averaging (SWA), a recent approach which averages weights along the trajectory of SGD with a modified learning rate schedule. We also propose fast-SWA, which further accelerates convergence by averaging multiple points within each cycle of a cyclical learning rate schedule. With weight averaging, we achieve the best known semi-supervised results on CIFAR-10 and CIFAR-100, over many different quantities of labeled training data. For example, we achieve 5.0% error on CIFAR-10 with only 4000 labels, compared to the previous best result in the literature of 6.3%.

Motivation & Objective

• Understand how loss geometry and training procedures affect consistency-based semi-supervised models.
• Investigate why SGD struggles to converge for consistency losses and how weight averaging stabilizes solutions.
• Propose and evaluate stochastic weight averaging (SWA) and fast-SWA to improve generalization.
• Demonstrate improved state-of-the-art results on CIFAR-10/100 with limited labeled data.
• Provide practical guidance for applying weight averaging to consistency-based methods.

Proposed method

• Analyze a simplified Pi model to show implicit regularization on input Jacobian and Hessian eigenvalues.
• Empirically compare training trajectories of Pi, Mean Teacher, and supervised models to show larger SGD steps and diverse solutions.
• Demonstrate that averaging weights (and predictions) across SGD iterates improves generalization.
• Introduce Stochastic Weight Averaging (SWA) and fast-SWA with cyclical learning rates and multi-point per cycle averaging.
• Apply SWA/fast-SWA to Pi and Mean Teacher on CIFAR-10/100 with varying labeled/unlabeled data and report improvements.

Experimental results

Research questions

• RQ1 How does the consistency loss affect the geometry of the loss landscape and model generalization?
• RQ2 Why do SGD trajectories under consistency-based methods remain diverse and what is the effect on test predictions?
• RQ3 Can weight averaging of SGD iterates (SWA/fast-SWA) outperform single-SGD solutions and standard ensembling in semi-supervised models?
• RQ4 Do Pi and Mean Teacher models achieve new state-of-the-art results on CIFAR-10/100 with limited labels when combined with SWA/fast-SWA?
• RQ5 How do additional unlabeled data (e.g., Tiny Images) affect performance with SWA/fast-SWA?

Key findings

• A simplified Pi model implicitly regularizes the input-output Jacobian norm and Hessian eigenvalues, promoting flatter solutions.
• Consistency-based models (Pi and MT) exhibit larger weight-space trajectories and higher prediction diversity than supervised models.
• Weight averaging of SGD iterates (SWA) or predictions significantly improves generalization, with larger gains for Pi/MT than for supervised training.
• Applying SWA and a faster variant (fast-SWA) substantially speeds up convergence and achieves top semi-supervised results on CIFAR-10/100, e.g., 5.0% error on CIFAR-10 with 4k labels.
• Fast-SWA converges faster than SWA and, in many setups, achieves equal or better performance with fewer cycles.
• Fast-SWA also improves a related domain adaptation task (CIFAR-10 to STL) from 19.9% to 16.8% error.