[Paper Review] RankingMatch: Delving into Semi-Supervised Learning with Consistency Regularization and Ranking Loss

Trung Quang Tran, Min‐Gu Kang+1 more|arXiv (Cornell University)|Oct 9, 2021

Domain Adaptation and Few-Shot Learning33 references4 citations

TL;DR

RankingMatch proposes a novel semi-supervised learning method that enhances consistency regularization by incorporating ranking-based loss to encourage similar model outputs for images of the same class. By introducing the computationally efficient BatchMean Triplet loss and applying it directly to model logits, RankingMatch achieves state-of-the-art performance, reaching 95.13% accuracy on CIFAR-10 with only 250 labels and 97.77% on SVHN with 1000 labels.

ABSTRACT

Semi-supervised learning (SSL) has played an important role in leveraging unlabeled data when labeled data is limited. One of the most successful SSL approaches is based on consistency regularization, which encourages the model to produce unchanged with perturbed input. However, there has been less attention spent on inputs that have the same label. Motivated by the observation that the inputs having the same label should have the similar model outputs, we propose a novel method, RankingMatch, that considers not only the perturbed inputs but also the similarity among the inputs having the same label. We especially introduce a new objective function, dubbed BatchMean Triplet loss, which has the advantage of computational efficiency while taking into account all input samples. Our RankingMatch achieves state-of-the-art performance across many standard SSL benchmarks with a variety of labeled data amounts, including 95.13% accuracy on CIFAR-10 with 250 labels, 77.65% accuracy on CIFAR-100 with 10000 labels, 97.76% accuracy on SVHN with 250 labels, and 97.77% accuracy on SVHN with 1000 labels. We also perform an ablation study to prove the efficacy of the proposed BatchMean Triplet loss against existing versions of Triplet loss.

Motivation & Objective

• To address the limitation of existing consistency regularization methods in semi-supervised learning, which focus only on perturbed versions of the same input and ignore inter-sample similarity within the same class.
• To improve model generalization by enforcing that images from the same class produce similar model outputs, even when not perturbations of each other.
• To develop a computationally efficient variant of triplet loss that considers all samples in a batch without the high cost of BatchAll or the complexity of BatchHard.
• To unify semi-supervised learning and metric learning by applying ranking losses directly to model logits rather than feature representations.
• To empirically validate the effectiveness of the proposed loss components and their impact on convergence and performance across diverse benchmarks and label budgets.

Proposed method

• Introduces RankingMatch, a semi-supervised learning framework that combines cross-entropy loss, consistency regularization (as in FixMatch), and a novel ranking loss component.
• Applies Triplet and Contrastive loss directly to the model's final logits (classification scores), not to learned feature embeddings, to enforce intra-class similarity at the output level.
• Proposes BatchMean Triplet loss as a new variant that computes the hardest positive and negative pairs across the entire batch, balancing computational efficiency and representativeness.
• Uses L2-normalization on logits to stabilize training and improve convergence, especially when using the BatchMean loss.
• Employs a two-augmentation strategy: weakly-augmented inputs generate pseudo-labels for strongly-augmented counterparts, enforcing consistency.
• Adapts the ranking loss to be differentiable and scalable, avoiding the memory and time overhead of BatchAll while outperforming BatchHard in accuracy.

Experimental results

Research questions

• RQ1 Can enforcing similarity among model outputs of same-class samples improve semi-supervised learning performance beyond consistency regularization alone?
• RQ2 Does applying ranking loss directly to logits (rather than features) yield better generalization and training stability?
• RQ3 Can a new triplet loss variant be designed that maintains the representativeness of BatchAll while achieving the efficiency of BatchHard?
• RQ4 How does the inclusion of L2-normalization affect the training dynamics and final accuracy of the ranking-based SSL method?
• RQ5 What is the relative contribution of the proposed BatchMean Triplet loss compared to existing variants (BatchAll, BatchHard) in terms of accuracy and computational cost?

Key findings

• RankingMatch achieves 95.13% top-1 accuracy on CIFAR-10 with only 250 labeled examples, outperforming prior state-of-the-art methods.
• On SVHN, RankingMatch reaches 97.77% accuracy with 1000 labels, demonstrating strong performance even at low label budgets.
• The proposed BatchMean Triplet loss reduces error rate by 50% and 75% compared to BatchHard and BatchAll variants on CIFAR-10 with 250 labels, respectively.
• Without L2-normalization, the BatchMean Triplet loss causes training divergence due to exploding gradients, highlighting its critical role in model stability.
• BatchMean Triplet loss is significantly more efficient than BatchAll, reducing average training time per epoch by over 125 seconds on SVHN and halving GPU memory usage.
• t-SNE visualization shows that RankingMatch produces more compact and well-separated class clusters in the logits space compared to FixMatch and MixMatch, suggesting improved decision boundaries.