[Paper Review] Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations

Mahmoud Assran, Nicolas Ballas+2 more|arXiv (Cornell University)|Jun 18, 2020

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes SuNCEt, a semi-supervised contrastive learning loss that integrates noise-contrastive estimation and neighborhood component analysis to leverage minimal labeled data during pre-training. By incorporating even a small amount of supervision early in training, SuNCEt accelerates convergence, enabling state-of-the-art ImageNet performance with less than half the pre-training compute compared to prior contrastive methods.

ABSTRACT

We investigate a strategy for improving the efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training. We propose a semi-supervised loss, SuNCEt, based on noise-contrastive estimation and neighbourhood component analysis, that aims to distinguish examples of different classes in addition to the self-supervised instance-wise pretext tasks. On ImageNet, we find that SuNCEt can be used to match the semi-supervised learning accuracy of previous contrastive approaches while using less than half the amount of pre-training and compute. Our main insight is that leveraging even a small amount of labeled data during pre-training, and not only during fine-tuning, provides an important signal that can significantly accelerate contrastive learning of visual representations. Our code is available online at github.com/facebookresearch/suncet.

Motivation & Objective

• To address the high computational cost of contrastive self-supervised pre-training in vision representation learning.
• To investigate whether incorporating even a small amount of labeled data during pre-training—rather than only during fine-tuning—can accelerate learning.
• To design a novel semi-supervised loss that improves convergence speed without increasing hyperparameter tuning or memory overhead.
• To achieve state-of-the-art semi-supervised accuracy on ImageNet with reduced pre-training compute and data efficiency.

Proposed method

• Proposes SuNCEt, a contrastive loss combining noise-contrastive estimation (NCE) and neighborhood component analysis (NCA) to distinguish between-class representations during pre-training.
• Uses a small labeled subset during pre-training to guide the contrastive objective, enabling faster convergence to high-quality representations.
• Integrates the SuNCEt loss with existing contrastive frameworks such as SimCLR and SwAV, maintaining compatibility with standard data augmentation and encoder architectures.
• Employs a normalized temperature-scaled cross-entropy loss for instance-wise discrimination, with positive pairs derived from augmented views and negatives from other images in the batch.
• Avoids the memory overhead of storing full embedding tensors by relying on NCE-based estimation instead of full NCA computation.
• Treats labeled examples as positive pairs during pre-training, allowing the model to learn class-level structure earlier in training.

Experimental results

Research questions

• RQ1 Can a small amount of labeled data during pre-training significantly accelerate contrastive representation learning?
• RQ2 Does integrating supervision earlier in the training pipeline reduce the required pre-training compute while maintaining or improving final accuracy?
• RQ3 How does SuNCEt compare to standard contrastive losses and other semi-supervised methods in terms of convergence speed and final performance?
• RQ4 Can SuNCEt be effectively combined with existing contrastive frameworks like SimCLR and SwAV without hyperparameter tuning?

Key findings

• SuNCEt achieves the same semi-supervised ImageNet top-5 accuracy as prior state-of-the-art contrastive methods while using less than half the pre-training compute and epochs.
• On ImageNet with 10% labeled data, SuNCEt combined with SwAV achieves state-of-the-art top-5 accuracy, outperforming FixMatch with RandAugment.
• The method requires no hyperparameter tuning and maintains consistent performance across different batch sizes and data augmentations.
• Even a small amount of labeled data during pre-training significantly accelerates convergence, reducing the need for extensive pre-training.
• SuNCEt outperforms baseline contrastive methods when used with larger batches and more positive samples per anchor, indicating scalability advantages.