[Paper Review] Semi-supervised Contrastive Learning with Similarity Co-calibration

Yuhang Zhang, Xiaopeng Zhang+4 more|arXiv (Cornell University)|May 16, 2021

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Semi-supervised Contrastive Learning (SsCL), a novel end-to-end framework that co-calibrates predictions between a cross-entropy branch and a contrastive loss branch to improve representation learning with limited labeled data. By interchanging pseudo-labels and similarity embeddings, SsCL achieves 60.2% top-1 accuracy on ImageNet with only 1% labeled data, significantly outperforming prior semi-supervised and self-supervised methods.

ABSTRACT

Semi-supervised learning acts as an effective way to leverage massive unlabeled data. In this paper, we propose a novel training strategy, termed as Semi-supervised Contrastive Learning (SsCL), which combines the well-known contrastive loss in self-supervised learning with the cross entropy loss in semi-supervised learning, and jointly optimizes the two objectives in an end-to-end way. The highlight is that different from self-training based semi-supervised learning that conducts prediction and retraining over the same model weights, SsCL interchanges the predictions over the unlabeled data between the two branches, and thus formulates a co-calibration procedure, which we find is beneficial for better prediction and avoid being trapped in local minimum. Towards this goal, the contrastive loss branch models pairwise similarities among samples, using the nearest neighborhood generated from the cross entropy branch, and in turn calibrates the prediction distribution of the cross entropy branch with the contrastive similarity. We show that SsCL produces more discriminative representation and is beneficial to few shot learning. Notably, on ImageNet with ResNet50 as the backbone, SsCL achieves 60.2% and 72.1% top-1 accuracy with 1% and 10% labeled samples, respectively, which significantly outperforms the baseline, and is better than previous semi-supervised and self-supervised methods.

Motivation & Objective

• To address the limited performance gain of cross-entropy-based semi-supervised learning on large-scale datasets like ImageNet due to noisy and unrepresentative labels.
• To improve representation learning by integrating class-specific priors into contrastive pretraining, avoiding task-agnostic self-supervised pretraining.
• To overcome local minima and model collapse in self-training by introducing a co-calibration mechanism between two loss branches.
• To enhance few-shot generalization through adaptive loss weighting, self-paced learning, and data mixing for class-specific sample enrichment.

Proposed method

• Introduces a co-calibration mechanism that alternately uses predictions from the cross-entropy branch to guide positive sample mining in the contrastive branch, and vice versa to refine predictions.
• Extends the contrastive loss to support multiple positive samples per anchor by pulling features toward class prototypes, enhancing discriminative representation.
• Applies a self-paced learning strategy that adaptively adjusts loss weights based on similarity confidence, reducing noise impact during early training.
• Employs a data mix strategy that combines ground-truth and similar samples during training to enrich class-specific features in a robust, smooth manner.
• Jointly optimizes cross-entropy and contrastive losses in an end-to-end fashion, with learnable loss weights (λ_ctr, λ_cls) for dynamic balance.
• Uses temperature scaling (τ) and margin-based contrastive loss (m) to stabilize training and improve feature clustering.

Figure 1: Comparisons between cross-entropy based semi-supervised learning and our proposed SsCL. The highlight is that we co-calibrate the predictions over the two branches and jointly train the two objectives in an end-to-end way.

Experimental results

Research questions

• RQ1 Can co-calibrating predictions between cross-entropy and contrastive learning branches improve generalization in semi-supervised learning?
• RQ2 Does incorporating class-specific priors into contrastive learning lead to better representations than task-agnostic self-supervised pretraining?
• RQ3 Can the co-calibration mechanism mitigate local minima and model collapse in self-training-based semi-supervised learning?
• RQ4 How does the proposed method perform under extreme label scarcity, such as 1% or 10% labeled data on ImageNet?
• RQ5 To what extent does the integration of contrastive learning improve few-shot transfer learning performance?

Key findings

• On ImageNet with ResNet-50, SsCL achieves 60.2% top-1 accuracy using only 1% labeled data, significantly outperforming existing semi-supervised and self-supervised methods.
• With 10% labeled data, SsCL attains 72.1% top-1 accuracy on ImageNet, demonstrating strong performance even under moderate label scarcity.
• In linear evaluation transfer learning, SsCL surpasses self-supervised baselines (SimCLR, MoCo v2) on all four benchmark datasets (CIFAR-10/100, FGVC-Aircraft, Stanford Cars).
• In fine-tuning transfer learning, SsCL exceeds even the supervised baseline on most datasets, indicating strong transferability of learned representations.
• t-SNE visualization confirms that SsCL features are more compact and discriminative than MoCo v2, though slightly less than fully supervised features.
• Ablation studies show that co-calibration and adaptive loss weighting significantly improve performance, especially in early training stages with noisy predictions.

Figure 2: An overview of our proposed semi-supervised contrastive learning framework. Different from traditional contrastive learning in unsupervised paradigm, the few shot labeled data is utilized for contrastive learning, which targets at learning more discriminative representation in an end-to-en