[論文レビュー] SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning

Hao Chen, Ran Tao+7 more|arXiv (Cornell University)|Jan 26, 2023

Domain Adaptation and Few-Shot Learning被引用数 89

ひとこと要約

SoftMatch は、SSL における擬似ラベルの量と質を高い状態に保つための、トランケーションされた Gaussian weighting と Uniform Alignment を導入し、未ラベルデータをより適切に活用することで学習を改善します。

ABSTRACT

The critical challenge of Semi-Supervised Learning (SSL) is how to effectively leverage the limited labeled data and massive unlabeled data to improve the model's generalization performance. In this paper, we first revisit the popular pseudo-labeling methods via a unified sample weighting formulation and demonstrate the inherent quantity-quality trade-off problem of pseudo-labeling with thresholding, which may prohibit learning. To this end, we propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training, effectively exploiting the unlabeled data. We derive a truncated Gaussian function to weight samples based on their confidence, which can be viewed as a soft version of the confidence threshold. We further enhance the utilization of weakly-learned classes by proposing a uniform alignment approach. In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.

研究の動機と目的

• Formally define the quantity and quality of pseudo-labels in SSL within a unified sample weighting framework.
• Identify the limitations of hard-thresholded pseudo-labeling that cause a quantity-quality trade-off.
• Propose SoftMatch with adaptive, truncated Gaussian weighting to improve pseudo-label utilization while maintaining quality.
• Introduce Uniform Alignment to address class-imbalance in pseudo-labels and stabilize learning.
• Demonstrate state-of-the-art or competitive results on image, text, and long-tailed SSL benchmarks.

提案手法

• Reframe SSL unsupervised loss as a weighted cross-entropy where weights lambda(p) depend on the model’s confidence.
• Derive a truncated Gaussian weighting function for lambda(p) based on the current training iteration mean mu_t and variance sigma_t of the max-class confidence.
• Estimate mu_t and sigma_t via EMA from historical predictions and adapt weights during training.
• Introduce Uniform Alignment to normalize predictions by a uniform target distribution and compute per-sample weights from the normalized predictions.
• Combine the Gaussian weighting with Uniform Alignment to form the final lambda(p) used in the unsupervised loss.
• Provide an algorithmic overview and empirical validation across image and text tasks.

Figure 4: Sample weighting function visualization

実験結果

リサーチクエスチョン

• RQ1 How can pseudo-labels be weighted to balance their quantity and quality during SSL training?
• RQ2 Can a data-driven, adaptive weighting scheme outperform fixed or threshold-based pseudo-labeling in diverse domains?
• RQ3 Does aligning pseudo-label distributions toward uniform class usage improve robustness under long-tailed or imbalanced data?
• RQ4 What is the impact of updating weighting parameters from historical predictions on SSL performance?
• RQ5 How does SoftMatch perform relative to state-of-the-art SSL methods on image and text benchmarks?

主な発見

• SoftMatch achieves strong improvements across image and text classification benchmarks compared to prior SSL methods.
• SoftMatch surpasses FixMatch by 1.48% on SVHN with 40 labels.
• SoftMatch outperforms FlexMatch on CIFAR-100 with 400 labels, STL-10 with 40 labels, and ImageNet with 10% labels by margins of 7.73%, 2.84%, and 1.33%, respectively.
• SoftMatch demonstrates robustness to long-tailed distributions in imbalanced SSL settings, showing best test error across all long-tailed configurations examined.
• Uniform Alignment helps balance pseudo-labels across classes, benefiting learning in imbalanced SSL scenarios.
• Ablation studies indicate Gaussian-based weighting with adaptive mu_t, sigma_t and Uniform Alignment yield the most consistent gains over alternatives.

(a) FixMatch