[Paper Review] CLS: Cross Labeling Supervision for Semi-Supervised Learning

Yao Yao, Junyi Shen+3 more|arXiv (Cornell University)|Feb 17, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Cross Labeling Supervision (CLS), a semi-supervised learning framework that enhances pseudo-labeling by introducing complementary labels and co-training between two independently initialized networks to mitigate confirmation bias. By assigning adaptive sample weights based on prediction confidence and enabling mutual supervision, CLS achieves state-of-the-art performance on CIFAR-10 and CIFAR-100, outperforming existing methods including FixMatch.

ABSTRACT

It is well known that the success of deep neural networks is greatly attributed to large-scale labeled datasets. However, it can be extremely time-consuming and laborious to collect sufficient high-quality labeled data in most practical applications. Semi-supervised learning (SSL) provides an effective solution to reduce the cost of labeling by simultaneously leveraging both labeled and unlabeled data. In this work, we present Cross Labeling Supervision (CLS), a framework that generalizes the typical pseudo-labeling process. Based on FixMatch, where a pseudo label is generated from a weakly-augmented sample to teach the prediction on a strong augmentation of the same input sample, CLS allows the creation of both pseudo and complementary labels to support both positive and negative learning. To mitigate the confirmation bias of self-labeling and boost the tolerance to false labels, two different initialized networks with the same structure are trained simultaneously. Each network utilizes high-confidence labels from the other network as additional supervision signals. During the label generation phase, adaptive sample weights are assigned to artificial labels according to their prediction confidence. The sample weight plays two roles: quantify the generated labels' quality and reduce the disruption of inaccurate labels on network training. Experimental results on the semi-supervised classification task show that our framework outperforms existing approaches by large margins on the CIFAR-10 and CIFAR-100 datasets.

Motivation & Objective

• To address confirmation bias in self-training-based semi-supervised learning, where inaccurate pseudo-labels degrade model performance.
• To improve robustness to noisy artificial labels by introducing complementary labels for negative learning.
• To reduce reliance on high-confidence pseudo-labels through co-training between two independently initialized networks.
• To enhance generalization by assigning adaptive sample weights to artificial labels based on prediction confidence.
• To achieve state-of-the-art performance on standard semi-supervised image classification benchmarks.

Proposed method

• CLS extends FixMatch by introducing two parallel, independently initialized networks that generate both pseudo-labels and complementary labels for each other.
• Each network uses high-confidence predictions from the other network as additional supervision signals, enabling co-training and reducing confirmation bias.
• Complementary labels are generated by taking the top-k classes with lowest confidence from the other network’s output, supporting negative learning.
• Adaptive sample weights are assigned to artificial labels based on their prediction confidence, reducing the impact of low-confidence or incorrect labels.
• The framework combines consistency regularization and pseudo-labeling, with strong augmentations used for training and weak augmentations for label generation.
• Training alternates between updating both networks using a combination of labeled data, pseudo-labels, and complementary labels with confidence-based weighting.

Experimental results

Research questions

• RQ1 Can mutual supervision between two independently trained networks reduce confirmation bias in semi-supervised learning?
• RQ2 How does the inclusion of complementary labels improve model robustness to noisy pseudo-labels?
• RQ3 To what extent does adaptive sample weighting based on prediction confidence enhance training stability and performance?
• RQ4 Does co-training with complementary labels lead to better generalization than standard pseudo-labeling in low-data regimes?
• RQ5 How does CLS compare to state-of-the-art methods like FixMatch and MixMatch on standard benchmarks?

Key findings

• CLS achieves state-of-the-art performance on CIFAR-10, outperforming FixMatch by a significant margin under standard semi-supervised learning settings.
• On CIFAR-100, CLS achieves superior accuracy compared to existing methods, demonstrating strong generalization in high-dimensional label spaces.
• The use of complementary labels and co-training significantly reduces the negative impact of low-confidence pseudo-labels, improving robustness.
• Adaptive sample weighting based on confidence effectively suppresses the influence of inaccurate labels during training.
• The framework’s dual-network co-training mechanism provides a regularizing effect, stabilizing learning and improving convergence.
• Empirical results confirm that CLS consistently improves performance across multiple runs and data splits, indicating high reliability.