[Paper Review] FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

Kihyuk Sohn, David Berthelot+7 more|arXiv (Cornell University)|Jan 21, 2020

Domain Adaptation and Few-Shot Learning60 references2,264 citations

TL;DR

FixMatch introduces a simple SSL algorithm that uses weak augmentation to generate pseudo-labels, then trains on strongly augmented versions to match those labels; it achieves state-of-the-art performance across standard SSL benchmarks, including CIFAR-10 with 250 labels and as few as 4 labels per class.

ABSTRACT

Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model's predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -- just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch's success. We make our code available at https://github.com/google-research/fixmatch.

Motivation & Objective

• Motivate SSL as a practical approach to leverage unlabeled data with minimal labeling effort.
• Propose a simple SSL algorithm that blends pseudo-labeling with consistency regularization.
• Show that strong augmentation paired with thresholded pseudo-labeling yields strong performance across benchmarks.
• Assess key factors influencing SSL performance, including augmentation strategies and optimizer choices.

Proposed method

• Generate pseudo-labels from weakly-augmented unlabeled images using model predictions.
• Retain pseudo-labels only if the maximum class probability exceeds a threshold τ.
• Train on strongly-augmented versions of the same unlabeled image to match the pseudo-label via cross-entropy loss.
• Combine the unlabeled loss with a standard supervised loss on labeled data (ℓs + λuℓu).
• Use weak augmentation for label prediction and strong augmentation (RandAugment/CTAugment with Cutout) for training.
• Report extensive ablations on factors like thresholding, sharpening, augmentation strategy, and regularization.

Experimental results

Research questions

• RQ1 Does a simpler combination of pseudo-labeling and consistency regularization outperform more complex SSL methods?
• RQ2 How do augmentation choices, thresholding, and optimization settings affect FixMatch performance in low-label regimes?
• RQ3 Can FixMatch achieve state-of-the-art results across standard SSL benchmarks with minimal hyperparameter tuning?

Key findings

• FixMatch achieves state-of-the-art results on CIFAR-10 with 250 labels (94.93% accuracy) and with 4 labels per class (88.61% accuracy).
• The method remains strong across CIFAR-10/100, SVHN, STL-10, and ImageNet, often outperforming prior SSL methods while remaining simpler.
• Threshold-based pseudo-labeling (τ = 0.95) and the use of strong augmentation are crucial for performance; sharpening alone does not improve results when thresholding is used.
• Regularization choices (weight decay) and SGD optimization are important factors influencing SSL success, even with FixMatch’s simplicity.
• Extensions from related SSL work (e.g., Distribution Alignment) can further improve FixMatch performance, though the core approach is simple and effective.