[论文解读] No Free Lunch in Self Supervised Representation Learning

Ihab Bendidi, Adrien Bardes+4 more|arXiv (Cornell University)|Apr 23, 2023

Cell Image Analysis Techniques被引用 9

一句话总结

本论文表明，在自监督表征学习（SSRL）中，数据增强的选择、强度与组合构成一种弱监督，偏向类别级别结果与下游任务，且在领域相关性上存在显著影响，尤其在显微镜图像中，领域专业知识可以显著提升性能。

ABSTRACT

Self-supervised representation learning in computer vision relies heavily on hand-crafted image transformations to learn meaningful and invariant features. However few extensive explorations of the impact of transformation design have been conducted in the literature. In particular, the dependence of downstream performances to transformation design has been established, but not studied in depth. In this work, we explore this relationship, its impact on a domain other than natural images, and show that designing the transformations can be viewed as a form of supervision. First, we demonstrate that not only do transformations have an effect on downstream performance and relevance of clustering, but also that each category in a supervised dataset can be impacted in a different way. Following this, we explore the impact of transformation design on microscopy images, a domain where the difference between classes is more subtle and fuzzy than in natural images. In this case, we observe a greater impact on downstream tasks performances. Finally, we demonstrate that transformation design can be leveraged as a form of supervision, as careful selection of these by a domain expert can lead to a drastic increase in performance on a given downstream task.

研究动机与目标

• 研究变换设计如何影响 SSRL 在类别级别的性能。
• 评估增强选择对下游任务（如聚类与分类）的影响。
• 考察自然图像与显微镜图像在增强效应上的差异。
• 证明领域专家选择的增强在具有挑战性的领域中可以显著改善 SSRL 结果。

提出的方法

• Systematically vary transformation intensities (amplitude and probability) across common augmentations in SSRL with ResNet18 on CIFAR-10/100 and ImageNet-100.
• Train multiple SSRL methods (Barlow Twins, MoCo v2, BYOL, SimCLR, VICReg) under varied augmentations.
• Quantify class-level performance changes and compute inter-class bias via correlations of per-class accuracies under different augmentations.
• Apply MoCo v2 with VGG-based encoders to MNIST to analyze how different transformation sets affect clustering quality (Silhouette, AMI) and linear evaluation.
• Study microscopy images from BBBC021v1 with VGG13 and MoCo v2 to evaluate how augmentation choices influence AMI-based clustering of cell phenotypes under subtle differences.
• Demonstrate domain-expert augmentation design can surpass pretrained supervised baselines in biological datasets.

Figure 1: A t-SNE projection of the ten-class clustering of the MNIST dataset (LeCun et al., 1998 ) performed on two representations obtained from two self-supervised trainings of the same model using MoCo V2 (Chen et al., 2020b ) , with the sole distinction being the selection of transformations em

实验结果

研究问题

• RQ1 变换强度或组成的变化是否会在 SSRL 表征中引入跨类别偏差？
• RQ2 增强选择 how influence 下游任务如聚类和线性评估在标准基准上的表现？
• RQ3 在显微镜图像等领域是否存在增强调制的特定影响，尤其是在类别差异微妙的情况下？
• RQ4 领域专家选择的增强是否能够在超越标准预训练模型的情况下显著改进 SSRL 表征？

主要发现

• 即使整体准确性保持稳定，增强参数也会导致每类别准确率发生显著变化。
• 某些类别在特定增强参数下受益或受损，表明存在类别间偏差。
• 不同下游任务（如聚类与线性准确率）对增强设计与组合的响应不同。
• 在显微镜数据中，变换选择影响更大，某些增强集在区分困难的情况下的 AMI 得分甚至可与预训练的 ResNet101 相当。
• 领域专家可设计的增强组合能够在聚类和表型分离的下游任务中超越预训练的有监督模型。

Figure 2: Inter-class accuracy results for Resnet18 architectures trained with various SSRL methods on the benchmark datasets Cifar10, Cifar100 and Imagenet100, as the parameters of different image transformations are varied. Each dot and associated error bar reflects the mean and standard deviation