[Paper Review] A Cookbook of Self-Supervised Learning

Randall Balestriero, Mark Ibrahim+17 more|arXiv (Cornell University)|Apr 24, 2023

Machine Learning and Data ClassificationComputer Science161 citations

TL;DR

This paper presents a cookbook-style guide to self-supervised learning (SSL), categorizing SSL families, summarizing theory, and detailing practical training and deployment recipes to lower barriers to entry.

ABSTRACT

Self-supervised learning, dubbed the dark matter of intelligence, is a promising path to advance machine learning. Yet, much like cooking, training SSL methods is a delicate art with a high barrier to entry. While many components are familiar, successfully training a SSL method involves a dizzying set of choices from the pretext tasks to training hyper-parameters. Our goal is to lower the barrier to entry into SSL research by laying the foundations and latest SSL recipes in the style of a cookbook. We hope to empower the curious researcher to navigate the terrain of methods, understand the role of the various knobs, and gain the know-how required to explore how delicious SSL can be.

Motivation & Objective

• Lower the barrier to entry into SSL research by providing foundations and recipes.
• Unify diverse SSL methods under a common vocabulary and theoretical perspective.
• Categorize SSL into core families and summarize their connections and differences.
• Offer practical guidance on data, architectures, hyperparameters, evaluation, and deployment of SSL models.

Proposed method

• Describe the fundamental SSL families and provide a unified theoretical perspective.
• Explain the origins and evolution of SSL and the key ideas behind each family.
• Summarize practical training considerations such as data augmentation, projectors, and teacher-student mechanisms.
• Discuss evaluation strategies with and without labels and how to measure downstream performance.
• Provide guidance on speedups, distributed training, and adaptations to non-image domains and multimodal data.

Experimental results

Research questions

• RQ1 What unifying framework can describe the main SSL families and their objectives?
• RQ2 What practical components and hyperparameters most influence SSL performance and reliability?
• RQ3 How can SSL be evaluated with and without labels, and beyond image classification?
• RQ4 What are effective strategies to accelerate SSL training and deployment in real-world settings?
• RQ5 How can SSL concepts extend beyond images to other data domains and modalities?

Key findings

• The work provides a unified view of SSL by organizing methods into four broad families: Deep Metric Learning, Self-Distillation, Canonical Correlation Analysis, and Masked Image Modeling.
• It highlights core techniques and loss paradigms (e.g., infoNCE, momentum encoders, predictors) that unify many SSL approaches.
• The cookbook outlines practical training recipes and common pitfalls to enable researchers to implement SSL effectively, including data augmentation, projector roles, and hyperparameter choices.
• It discusses evaluation strategies for SSL models both with and without labels and emphasizes broader evaluation beyond standard classification tasks.
• The document covers scalable training practices and speedups, such as distributed training and faster data pipelines, while also addressing extensions to other domains and modalities.