[Paper Review] Dash: Semi-Supervised Learning with Dynamic Thresholding

Yi Xu, Lei Shang+6 more|arXiv (Cornell University)|Sep 1, 2021

Machine Learning and Data Classification52 citations

TL;DR

Dash introduces a dynamic thresholding mechanism to select unlabeled data during SSL training, improving performance by adapting which pseudo-labeled examples are used at each iteration, with theoretical convergence guarantees.

ABSTRACT

While semi-supervised learning (SSL) has received tremendous attentions in many machine learning tasks due to its successful use of unlabeled data, existing SSL algorithms use either all unlabeled examples or the unlabeled examples with a fixed high-confidence prediction during the training progress. However, it is possible that too many correct/wrong pseudo labeled examples are eliminated/selected. In this work we develop a simple yet powerful framework, whose key idea is to select a subset of training examples from the unlabeled data when performing existing SSL methods so that only the unlabeled examples with pseudo labels related to the labeled data will be used to train models. The selection is performed at each updating iteration by only keeping the examples whose losses are smaller than a given threshold that is dynamically adjusted through the iteration. Our proposed approach, Dash, enjoys its adaptivity in terms of unlabeled data selection and its theoretical guarantee. Specifically, we theoretically establish the convergence rate of Dash from the view of non-convex optimization. Finally, we empirically demonstrate the effectiveness of the proposed method in comparison with state-of-the-art over benchmarks.

Motivation & Objective

• Motivate improving SSL by avoiding fixed high-confidence thresholds that can discard useful unlabeled data.
• Propose a dynamic thresholding framework (Dash) that selects unlabeled data at each iteration based on a decreasing loss threshold.
• Provide theoretical convergence guarantees for the Dash algorithm in a non-convex setting.
• Demonstrate empirical efficacy of Dash against state-of-the-art SSL methods on image classification benchmarks.

Proposed method

• Dash selects a subset of unlabeled data at each update by keeping examples with losses below a dynamic threshold rho_t.
• The threshold rho_t is set as rho_t = C * gamma^{-(t-1)} * rho_hat, decreasing over iterations.
• Initial warm-up stage trains on labeled data to estimate rho_hat; subsequent selection stage uses unlabeled data with pseudo labels from FixMatch.
• Stochastic gradients are computed using only unlabeled examples whose unsupervised loss f_u(w; xi^u) <= rho_t, combined with labeled data losses.
• Dash can be integrated with existing SSL pipelines like FixMatch and provides a non-asymptotic convergence guarantee under standard assumptions (PL condition).
• Theoretical results establish sample complexity and convergence rate, matching a supervised SGD-like rate under non-convex assumptions.

Experimental results

Research questions

• RQ1 Can we design an SSL algorithm that provably converges when unlabeled data come from a mixture of distributions?
• RQ2 Does dynamically selecting unlabeled data via a decreasing loss threshold improve SSL performance over fixed-threshold methods like FixMatch?
• RQ3 How should the dynamic threshold be constructed and estimated to balance inclusion of correct pseudo labels and exclusion of incorrect ones?
• RQ4 What are the theoretical convergence guarantees and sample complexity of such a dynamic-threshold SSL method?

Key findings

• Dash achieves non-asymptotic convergence guarantees for the proposed dynamic-threshold SSL under non-convex settings.
• Empirically, Dash outperforms several state-of-the-art SSL methods on standard image classification benchmarks (CIFAR-10, CIFAR-100, SVHN, STL-10) across various label regimes.
• Dash maintains more correct pseudo-labeled unlabeled examples early in training and reduces incorrect ones more aggressively in later epochs compared to fixed-threshold methods like FixMatch.
• Theoretical results show Dash’s convergence with high probability and provide a concrete sample complexity bound of O(1/epsilon).
• Experiments using different augmentation regimes (CTA, RA) demonstrate Dash’s compatibility and competitive gains with FixMatch-based pipelines.