[Paper Review] In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning

Mamshad Nayeem Rizve, Kévin Duarte+2 more|arXiv (Cornell University)|Jan 15, 2021

Text and Document Classification TechnologiesComputer Science242 citations

TL;DR

The paper defends pseudo-labeling by introducing an uncertainty-aware pseudo-label selection (UPS) framework that reduces label noise, enables negative labels for multi-label learning, and achieves strong SSL performance across CIFAR-10/100, UCF-101, and Pascal VOC.

ABSTRACT

The recent research in semi-supervised learning (SSL) is mostly dominated by consistency regularization based methods which achieve strong performance. However, they heavily rely on domain-specific data augmentations, which are not easy to generate for all data modalities. Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models; these predictions generate many incorrect pseudo-labels, leading to noisy training. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process. Furthermore, UPS generalizes the pseudo-labeling process, allowing for the creation of negative pseudo-labels; these negative pseudo-labels can be used for multi-label classification as well as negative learning to improve the single-label classification. We achieve strong performance when compared to recent SSL methods on the CIFAR-10 and CIFAR-100 datasets. Also, we demonstrate the versatility of our method on the video dataset UCF-101 and the multi-label dataset Pascal VOC.

Motivation & Objective

• Motivate pseudo-labeling as a general SSL approach that does not depend on domain-specific augmentations.
• Address poor calibration in networks that leads to high-confidence wrong pseudo-labels.
• Propose UPS to select low-uncertainty, high-confidence pseudo-labels and introduce negative pseudo-labels for multi-label learning.
• Demonstrate strong empirical results on CIFAR-10/100, UCF-101, and Pascal VOC using UPS.
• Show robustness across uncertainty estimation methods and discuss applicability to multiple modalities.

Proposed method

• Formalize pseudo-label generation from network predictions using hard thresholds (gamma) to create y-tilde.
• Introduce a subset selection vector g that uses confidence thresholds (tau_p, tau_n) to select positive/negative pseudo-labels.
• Incorporate uncertainty u(p) with thresholds (kappa_p, kappa_n) to form an uncertainty-aware selection (UPS) criterion.
• Train with a combination of labeled data and the selected subset of pseudo-labels using positive cross-entropy or negative learning when no positive labels are selected, plus a modified BCE when multi-label.
• Iteratively re-train with re-generated labels and reinitialize networks to limit error propagation (Algorithm 1).
• Demonstrate that UPS is modality-agnostic, compatible with various uncertainty estimation methods (MC-Dropout, SpatialDropout, DropBlock, Data Aug).

Experimental results

Research questions

• RQ1 Can pseudo-labeling approach achieve competitive SSL performance when augmented with uncertainty-aware pseudo-label selection?
• RQ2 How does the inclusion of uncertainty in pseudo-label selection affect calibration-related noise and generalization?
• RQ3 Does UPS extend naturally to multi-label and video domains via negative learning and uncertainty-based selection?
• RQ4 Is UPS robust to the choice of uncertainty estimation method and hyperparameters across multiple datasets?

Key findings

• UPS significantly reduces pseudo-label noise by using uncertainty-aware selection, achieving higher pseudo-label accuracy than conventional PL and confidence-based PL.
• In CIFAR-10/100, UPS attains competitive or superior error rates compared to state-of-the-art SSL methods across different label regimes and backbones.
• UPS outperforms baselines on UCF-101 (video domain) and Pascal VOC2007 (multi-label) even with limited labeled data.
• Ablation studies show that uncertainty-aware selection and negative learning jointly contribute to substantial error reductions, with UPS providing robustness to hyperparameters.
• UPS is compatible with multiple uncertainty estimation methods and does not rely on domain-specific augmentations, supporting broad modality applicability.