[Paper Review] No Free Lunch in Self Supervised Representation Learning

Ihab Bendidi, Adrien Bardes+4 more|arXiv (Cornell University)|Apr 23, 2023

Cell Image Analysis Techniques9 citations

TL;DR

This paper shows that the choice, intensity, and combination of data augmentations in self-supervised representation learning (SSRL) act as a form of weak supervision, biasing class-level outcomes and downstream tasks, with domain-dependent effects, notably in microscopy images where domain expertise can drastically improve performance.

ABSTRACT

Self-supervised representation learning in computer vision relies heavily on hand-crafted image transformations to learn meaningful and invariant features. However few extensive explorations of the impact of transformation design have been conducted in the literature. In particular, the dependence of downstream performances to transformation design has been established, but not studied in depth. In this work, we explore this relationship, its impact on a domain other than natural images, and show that designing the transformations can be viewed as a form of supervision. First, we demonstrate that not only do transformations have an effect on downstream performance and relevance of clustering, but also that each category in a supervised dataset can be impacted in a different way. Following this, we explore the impact of transformation design on microscopy images, a domain where the difference between classes is more subtle and fuzzy than in natural images. In this case, we observe a greater impact on downstream tasks performances. Finally, we demonstrate that transformation design can be leveraged as a form of supervision, as careful selection of these by a domain expert can lead to a drastic increase in performance on a given downstream task.

Motivation & Objective

• Investigate how transformation design affects SSRL performance at the class level.
• Assess the impact of augmentation choices on downstream tasks like clustering and classification.
• Examine the differences in augmentation effects between natural images and microscopy images.
• Demonstrate that domain-expert selection of augmentations can notably improve SSRL outcomes in challenging domains.

Proposed method

• Systematically vary transformation intensities (amplitude and probability) across common augmentations in SSRL with ResNet18 on CIFAR-10/100 and ImageNet-100.
• Train multiple SSRL methods (Barlow Twins, MoCo v2, BYOL, SimCLR, VICReg) under varied augmentations.
• Quantify class-level performance changes and compute inter-class bias via correlations of per-class accuracies under different augmentations.
• Apply MoCo v2 with VGG-based encoders to MNIST to analyze how different transformation sets affect clustering quality (Silhouette, AMI) and linear evaluation.
• Study microscopy images from BBBC021v1 with VGG13 and MoCo v2 to evaluate how augmentation choices influence AMI-based clustering of cell phenotypes under subtle differences.
• Demonstrate domain-expert augmentation design can surpass pretrained supervised baselines in biological datasets.

Figure 1: A t-SNE projection of the ten-class clustering of the MNIST dataset (LeCun et al., 1998 ) performed on two representations obtained from two self-supervised trainings of the same model using MoCo V2 (Chen et al., 2020b ) , with the sole distinction being the selection of transformations em

Experimental results

Research questions

• RQ1 Does varying augmentation intensity or composition induce inter-class biases in SSRL representations?
• RQ2 How do augmentation choices influence downstream tasks such as clustering and linear evaluation across standard benchmarks?
• RQ3 Are there domain-specific effects of augmentations, particularly in microscopy images where class differences are subtle?
• RQ4 Can domain-expert selection of augmentations meaningfully improve SSRL representations beyond standard pretrained models?

Key findings

• Augmentation parameters can cause meaningful shifts in per-class accuracy even when overall accuracy remains stable.
• Certain classes benefit or suffer under specific augmentation parameters, indicating inter-class bias.
• Different downstream tasks (e.g., clustering vs. linear accuracy) respond differently to augmentation design and compositions.
• In microscopy data, transformation choices have larger impacts, with some augmentation sets yielding AMI scores comparable to pretrained ResNet101 on challenging distinctions.
• Domain experts can craft augmentation combinations that outperform pretrained supervised models in clustering and downstream separation of phenotypes.

Figure 2: Inter-class accuracy results for Resnet18 architectures trained with various SSRL methods on the benchmark datasets Cifar10, Cifar100 and Imagenet100, as the parameters of different image transformations are varied. Each dot and associated error bar reflects the mean and standard deviation