[Paper Review] FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling

Bowen Zhang, Yidong Wang+5 more|arXiv (Cornell University)|Oct 15, 2021

Domain Adaptation and Few-Shot Learning37 references477 citations

TL;DR

FlexMatch integrates Curriculum Pseudo Labeling (CPL) with FixMatch to dynamically adjust per-class thresholds during SSL, boosting accuracy and convergence speed without extra parameters. CPL improves performance, especially with scarce labels, and can be applied to other SSL methods.

ABSTRACT

The recently proposed FixMatch achieved state-of-the-art results on most semi-supervised learning (SSL) benchmarks. However, like other modern SSL algorithms, FixMatch uses a pre-defined constant threshold for all classes to select unlabeled data that contribute to the training, thus failing to consider different learning status and learning difficulties of different classes. To address this issue, we propose Curriculum Pseudo Labeling (CPL), a curriculum learning approach to leverage unlabeled data according to the model's learning status. The core of CPL is to flexibly adjust thresholds for different classes at each time step to let pass informative unlabeled data and their pseudo labels. CPL does not introduce additional parameters or computations (forward or backward propagation). We apply CPL to FixMatch and call our improved algorithm FlexMatch. FlexMatch achieves state-of-the-art performance on a variety of SSL benchmarks, with especially strong performances when the labeled data are extremely limited or when the task is challenging. For example, FlexMatch achieves 13.96% and 18.96% error rate reduction over FixMatch on CIFAR-100 and STL-10 datasets respectively, when there are only 4 labels per class. CPL also significantly boosts the convergence speed, e.g., FlexMatch can use only 1/5 training time of FixMatch to achieve even better performance. Furthermore, we show that CPL can be easily adapted to other SSL algorithms and remarkably improve their performances. We open-source our code at https://github.com/TorchSSL/TorchSSL.

Motivation & Objective

• Motivate improvements over FixMatch by addressing fixed class-agnostic thresholds in SSL.
• Introduce Curriculum Pseudo Labeling (CPL) to leverage unlabeled data based on per-class learning status.
• Develop FlexMatch by integrating CPL with FixMatch without adding extra parameters or computations.
• Demonstrate CPL's effectiveness across standard SSL benchmarks and label-scarce scenarios.

Proposed method

• Define CPL to estimate per-class learning status using the number of unlabeled samples exceeding a fixed threshold.
• Compute per-class flexible thresholds T_t(c) by scaling the base threshold τ with a normalized learning-effect β_t(c).
• Update thresholds at each iteration without introducing new parameters or forward/backward computations.
• Apply a non-linear mapping M to β_t(c) to obtain T_t(c), using a convex function for better sensitivity.
• Incorporate a threshold warm-up to stabilize early training and reduce confirmation bias.
• Formulate the SSL loss as a combination of supervised loss L_s and unsupervised loss L_{u,t} with dynamic thresholds.

Experimental results

Research questions

• RQ1 Can per-class, time-varying thresholds improve unlabeled data utilization in SSL compared to a fixed global threshold?
• RQ2 Does CPL improve convergence speed and final accuracy across standard SSL benchmarks, especially with very limited labels?
• RQ3 Can CPL be effectively integrated withFixMatch and other SSL algorithms without adding extra trainable parameters or computational overhead?
• RQ4 What is the impact of threshold mapping functions and warm-up on performance?
• RQ5 How does CPL perform on large-scale and imbalanced datasets like ImageNet?

Key findings

• FlexMatch achieves state-of-the-art results on multiple SSL benchmarks, notably with extremely limited labels (e.g., CIFAR-100 with 4 labels per class, STL-10 with 40 labels).
• FlexMatch reduces error rates relative to FixMatch by 13.96% on CIFAR-100 and 18.96% on STL-10 under 4 labels per class.
• CPL accelerates convergence, enabling FlexMatch to reach competitive performance in less than 1/5 of FixMatch training time.
• CPL also improves other SSL methods (e.g., UDA, Pseudo-Labeling) without added computational cost.
• ImageNet experiments with 100 labels per class show FlexMatch outperforming FixMatch after 2^20 iterations, indicating scalability to large datasets.