[Paper Review] A Framework For Contrastive Self-Supervised Learning And Designing A New Approach

William Falcon, Kyunghyun Cho|arXiv (Cornell University)|Aug 31, 2020

Domain Adaptation and Few-Shot Learning30 references71 citations

TL;DR

The paper presents a five-part framework to unify CSL approaches (data augmentation, encoder, representation extraction, similarity, loss) and introduces YADIM, a robust CSL variant combining AMDIM and CPC ideas with standardized implementations.

ABSTRACT

Contrastive self-supervised learning (CSL) is an approach to learn useful representations by solving a pretext task that selects and compares anchor, negative and positive (APN) features from an unlabeled dataset. We present a conceptual framework that characterizes CSL approaches in five aspects (1) data augmentation pipeline, (2) encoder selection, (3) representation extraction, (4) similarity measure, and (5) loss function. We analyze three leading CSL approaches--AMDIM, CPC, and SimCLR--, and show that despite different motivations, they are special cases under this framework. We show the utility of our framework by designing Yet Another DIM (YADIM) which achieves competitive results on CIFAR-10, STL-10 and ImageNet, and is more robust to the choice of encoder and the representation extraction strategy. To support ongoing CSL research, we release the PyTorch implementation of this conceptual framework along with standardized implementations of AMDIM, CPC (V2), SimCLR, BYOL, Moco (V2) and YADIM.

Motivation & Objective

• Develop a unified five-component framework to characterize contrastive self-supervised learning (CSL) methods.
• Demonstrate that AMDIM, CPC, and SimCLR are special cases within the framework.
• Design and evaluate a new CSL variant (YADIM) that merges ideas from AMDIM and CPC.
• Show that YADIM achieves competitive performance and robustness across encoders and representations.
• Provide standardized, reproducible PyTorch implementations of CSL methods.

Proposed method

• Define a five-part CSL framework: data augmentation, encoder, representation extraction, similarity measure, and loss function.
• Analyze AMDIM, CPC, and SimCLR as special cases under the framework.
• Propose YADIM by merging CPC and AMDIM components and performing ablations to select robust design choices.
• Use NCE-based loss functions to train anchor–positive–negative triplets.
• Re-implement AMDIM, CPC, SimCLR, MoCo, and CMC in PyTorch Lightning to ensure consistent evaluation.
• Experiment with CIFAR-10, STL-10, and ImageNet to compare class separability and robustness across encoders.

Experimental results

Research questions

• RQ1 Can a unified framework meaningfully characterize major CSL approaches (AMDIM, CPC, SimCLR) as special cases?
• RQ2 Does a combined variant like YADIM maintain performance while increasing robustness to encoder choice and representation extraction strategy?
• RQ3 How do design choices in encoder architecture and representation extraction impact CSL performance across datasets?
• RQ4 What are the trade-offs between data augmentation pipelines in CSL when merging AMDIM and CPC ideas?
• RQ5 Do standardized, reproducible implementations affect cross-method comparability for CSL?

Key findings

• A unified framework shows AMDIM, CPC, and SimCLR are closely related variants under five design aspects.
• YADIM achieves competitive results on CIFAR-10, STL-10, and ImageNet, with improved robustness to encoder choice and simpler representation extraction.
• Encoder robustness: YADIM and CPC are less sensitive to encoder selection than AMDIM on CIFAR-10.
• Representation extraction: AMDIM’s multi-scale comparisons are more sensitive to the chosen extraction strategy than YADIM.
• YADIM’s final design uses a union data augmentation pipeline, wide ResNet-34 encoder, last-layer feature map comparisons, and a simple dot-product similarity with NCE loss.
• Extensive ablations show YADIM’s performance is stable across encoder variants and comparison strategies.