[Paper Review] FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning

Yidong Wang, Hao Chen+10 more|arXiv (Cornell University)|May 15, 2022

Domain Adaptation and Few-Shot Learning139 citations

TL;DR

FreeMatch introduces self-adaptive thresholding (SAT) and self-adaptive class fairness (SAF) to SSL, jointly adjusting per-class and global confidence thresholds based on learning status, yielding superior results especially with very scarce labels.

ABSTRACT

Semi-supervised Learning (SSL) has witnessed great success owing to the impressive performances brought by various methods based on pseudo labeling and consistency regularization. However, we argue that existing methods might fail to utilize the unlabeled data more effectively since they either use a pre-defined / fixed threshold or an ad-hoc threshold adjusting scheme, resulting in inferior performance and slow convergence. We first analyze a motivating example to obtain intuitions on the relationship between the desirable threshold and model's learning status. Based on the analysis, we hence propose FreeMatch to adjust the confidence threshold in a self-adaptive manner according to the model's learning status. We further introduce a self-adaptive class fairness regularization penalty to encourage the model for diverse predictions during the early training stage. Extensive experiments indicate the superiority of FreeMatch especially when the labeled data are extremely rare. FreeMatch achieves 5.78%, 13.59%, and 1.28% error rate reduction over the latest state-of-the-art method FlexMatch on CIFAR-10 with 1 label per class, STL-10 with 4 labels per class, and ImageNet with 100 labels per class, respectively. Moreover, FreeMatch can also boost the performance of imbalanced SSL. The codes can be found at https://github.com/microsoft/Semi-supervised-learning.

Motivation & Objective

• Motivate threshold design in SSL to reflect model learning status and data difficulty.
• Propose SAT to dynamically set global and class-specific thresholds without manual tuning.
• Introduce SAF to encourage diverse per-class predictions, improving learning with little labeled data.
• Demonstrate strong empirical gains across standard SSL benchmarks, especially under ultra-low-label regimes.

Proposed method

• SAT estimates a global threshold as EMA of unlabeled data confidence to reflect learning status.
• Local class thresholds are computed as EMA-based class confidences and combined with the global threshold via MaxNorm scaling.
• The unsupervised loss uses only unlabeled samples whose predicted confidence exceeds the self-adaptive threshold.
• SAF regularizes predictions to be diverse by aligning the batch’s predicted class distribution with a self-adaptive target distribution based on EMA statistics and histograms.
• The overall training objective combines supervised loss, unsupervised loss, and fairness loss: L = Ls + w_u Lu + w_f Lf.

Experimental results

Research questions

• RQ1 How should confidence thresholds be adapted during SSL training to balance data utilization and avoid confirmation bias?
• RQ2 Can per-class (local) thresholds improve SSL performance over a single global threshold, especially with imbalanced or hard-to-classify classes?
• RQ3 Does incorporating a self-adaptive class fairness objective improve learning when labeled data are extremely scarce?

Key findings

• FreeMatch achieves state-of-the-art or competitive results across SSL benchmarks, notably reducing error rates when labels are extremely scarce (e.g., CIFAR-10 with 1 label per class, STL-10 with 40 labels, ImageNet with 100 labels per class).
• Global and class-specific EMA-based thresholds yield faster convergence and better pseudo-label quality by reflecting learning status.
• SAF improves diversity of predictions and accelerates convergence, particularly in barely supervised settings.
• On ImageNet with 100 labels per class, FreeMatch outperforms FlexMatch by about 1.28 percentage points in Top-1 error.
• FreeMatch demonstrates robust improvements across CIFAR-10/100, SVHN, STL-10, and ImageNet compared to prior SSL methods.