[Paper Review] A Review on Discriminative Self-supervised Learning Methods in Computer Vision

Nikolaos Giakoumoglou, Tania Stathaki+1 more|arXiv (Cornell University)|May 8, 2024

Face and Expression Recognition4 citations

TL;DR

This paper provides a comprehensive review of discriminative self-supervised learning methods in computer vision, categorizing them into contrastive, distillation, feature decorrelation, and clustering approaches. It evaluates their performance on ImageNet, showing that methods like MoCo v3, DINO, and Mugs achieve top-1 accuracies above 80% under linear evaluation with ViT backbones, demonstrating state-of-the-art performance without human-annotated labels.

ABSTRACT

Self-supervised learning (SSL) has rapidly emerged as a transformative approach in computer vision, enabling the extraction of rich feature representations from vast amounts of unlabeled data and reducing reliance on costly manual annotations. This review presents a comprehensive analysis of discriminative SSL methods, which focus on learning representations by solving pretext tasks that do not require human labels. The paper systematically categorizes discriminative SSL approaches into five main groups: contrastive methods, clustering methods, self-distillation methods, knowledge distillation methods, and feature decorrelation methods. For each category, the review details the underlying principles, architectural components, loss functions, and representative algorithms, highlighting their unique mechanisms and contributions to the field. Extensive comparative evaluations are provided, including linear and semi-supervised protocols on standard benchmarks such as ImageNet, as well as transfer learning performance across diverse downstream tasks. The review also discusses theoretical foundations, scalability, efficiency, and practical challenges, such as computational demands and accessibility. By synthesizing recent advancements and identifying key trends, open challenges, and future research directions, this work serves as a valuable resource for researchers and practitioners aiming to leverage discriminative SSL for robust and generalizable computer vision models.

Motivation & Objective

• To provide a systematic review of discriminative self-supervised learning methods in computer vision, focusing on contrastive, distillation, feature decorrelation, and clustering approaches.
• To analyze the evolution and current state of self-supervised learning by examining key methodological components and training objectives.
• To benchmark and compare the performance of these methods on the standard ImageNet classification benchmark under linear evaluation and semi-supervised fine-tuning protocols.
• To highlight the effectiveness of self-supervised representations in matching or surpassing supervised pretraining, especially with limited labeled data.
• To serve as a comprehensive guide for researchers to understand methodological differences, similarities, and performance trade-offs in self-supervised learning.

Proposed method

• The paper categorizes self-supervised learning methods into four main groups: contrastive learning using instance discrimination with positive and negative views, self-distillation and knowledge distillation using momentum or teacher-student frameworks, feature decorrelation via invariance and redundancy reduction losses (e.g., Barlow Twins, VICReg), and clustering-based methods using momentum updates and predictive heads.
• It evaluates methods using standard ImageNet linear evaluation protocol, where a linear classifier is trained on frozen features extracted from pre-trained models.
• Semi-supervised fine-tuning is assessed using 1% and 10% of ImageNet’s labeled data, following standard protocols from SimCLR and BYOL.
• The analysis includes both CNN-based backbones (e.g., ResNet, RegNet) and Vision Transformers (ViT, Swin), with performance reported in top-1 and top-5 accuracy.
• Key techniques include multi-crop augmentation, momentum encoders, memory banks, and contrastive objectives such as InfoNCE, while methods like MoCo v3 and DINO use momentum updates and momentum keys.
• The study compares methods across multiple backbones and data regimes, including supervised pretraining as an upper bound for performance comparison.

(a) SimCLR [ 36 ]

Experimental results

Research questions

• RQ1 How do different discriminative self-supervised learning methods—contrastive, distillation, decorrelation, and clustering—compare in performance on ImageNet under linear evaluation?
• RQ2 What is the impact of backbone architecture (CNN vs. Vision Transformer) on the performance of self-supervised methods?
• RQ3 How effective are self-supervised models when fine-tuned with only 1% or 10% of ImageNet’s labeled data, and which methods generalize best under low-data regimes?
• RQ4 What are the key architectural and loss design choices that lead to state-of-the-art performance in self-supervised representation learning?
• RQ5 How do methods like MoCo v3, DINO, and Mugs achieve high accuracy without human-annotated labels, and what distinguishes their training objectives?

Key findings

• DINO with a ViT-Base/16 backbone achieved 80.1% top-1 accuracy under linear evaluation, approaching supervised pretraining (76.5%)
• Mugs with a ViT-Base/16 backbone reached 82.1% top-1 accuracy, outperforming all other methods in the study
• MoCo v3 with ViT-Base/16 achieved 76.5% top-1 accuracy, demonstrating strong performance with contrastive learning
• TWIST with ViT-Base/16 achieved 78.4% top-1 accuracy, showing strong results in feature decorrelation-based methods
• SMoG with Swin-T achieved 77.7% top-1 accuracy, indicating strong performance in clustering-based self-supervised learning
• Under 1% supervised fine-tuning, C-BYOL achieved 60.6% top-1 accuracy, significantly outperforming supervised pretraining with only 25.4% top-1 accuracy on the same split

(b) MoCo [ 8 ]