[Paper Review] On the surprising similarities between supervised and self-supervised models

Robert Geirhos, Kantharaju Narayanappa+4 more|arXiv (Cornell University)|Oct 16, 2020

Domain Adaptation and Few-Shot LearningComputer Science24 references19 citations

TL;DR

This study compares self-supervised and supervised deep learning models against human psychophysical data across 15 generalization datasets (130K trials), revealing that despite different training objectives, self-supervised models exhibit strikingly similar behavioral patterns to supervised models—particularly in poor noise robustness (except SimCLR), non-human error patterns, and strong texture bias—suggesting current self-supervised approaches do not yet emulate human-like visual recognition.

ABSTRACT

How do humans learn to acquire a powerful, flexible and robust representation of objects? While much of this process remains unknown, it is clear that humans do not require millions of object labels. Excitingly, recent algorithmic advancements in self-supervised learning now enable convolutional neural networks (CNNs) to learn useful visual object representations without supervised labels, too. In the light of this recent breakthrough, we here compare self-supervised networks to supervised models and human behaviour. We tested models on 15 generalisation datasets for which large-scale human behavioural data is available (130K highly controlled psychophysical trials). Surprisingly, current self-supervised CNNs share four key characteristics of their supervised counterparts: (1.) relatively poor noise robustness (with the notable exception of SimCLR), (2.) non-human category-level error patterns, (3.) non-human image-level error patterns (yet high similarity to supervised model errors) and (4.) a bias towards texture. Taken together, these results suggest that the strategies learned through today's supervised and self-supervised training objectives end up being surprisingly similar, but distant from human-like behaviour. That being said, we are clearly just at the beginning of what could be called a self-supervised revolution of machine vision, and we are hopeful that future self-supervised models behave differently from supervised ones, and---perhaps---more similar to robust human object recognition.

Motivation & Objective

• To investigate whether self-supervised models behave more like humans than supervised models in visual recognition tasks.
• To compare the generalization behavior of self-supervised and supervised models under various image distortions and perturbations.
• To assess whether self-supervised models exhibit human-like invariances or biases in object recognition, particularly regarding shape vs. texture.
• To evaluate the consistency of model errors with human error patterns across diverse visual stimuli.
• To determine whether contrastive self-supervised learning leads to fundamentally different inductive biases than supervised learning.

Proposed method

• Trained 8 self-supervised models (PIRL, MoCo, MoCoV2, InfoMin, InsDis, SimCLR-x1/x2/x4) and 24 supervised models (e.g., ResNet, VGG, DenseNet) using ResNet-50 architecture.
• Fine-tuned a linear classifier on top of each pre-trained self-supervised representation using ImageNet for evaluation.
• Evaluated models on 12 image degradation types (e.g., noise, rotation, phase noise) and texture-vs-shape conflict datasets.
• Used large-scale human psychophysical data (130K trials) from [4, 5] as a benchmark for human-like behavior.
• Applied error consistency analysis to compare model and human error patterns across image-level and category-level predictions.
• Quantified texture bias using a texture-vs-shape cue conflict dataset where shape and texture belong to different categories.

Experimental results

Research questions

• RQ1 Do self-supervised models generalize better than supervised models under image distortions and degradations?
• RQ2 Do self-supervised models make similar errors to humans or to supervised models on out-of-distribution stimuli?
• RQ3 Do self-supervised models exhibit a bias toward texture or shape in object recognition, and how does this compare to human perception?
• RQ4 To what extent do self-supervised and supervised models share similar inductive biases in their decision-making processes?
• RQ5 Is SimCLR, which uses specific data augmentations, an exception in robustness and human-likeness compared to other self-supervised methods?

Key findings

• Most self-supervised models show poor robustness to image distortions, similar to supervised models, with the exception of SimCLR, which demonstrates significantly better noise robustness.
• Self-supervised models share strong error consistency with supervised models—especially in image-level error patterns—indicating similar underlying inductive biases.
• Category-level error patterns in self-supervised models diverge from human behavior, showing no alignment with human perceptual grouping or categorization.
• All tested self-supervised models exhibit a strong texture bias, with SimCLR showing the least bias (58.3–61.2% texture decisions), still comparable to supervised Inception-V3 (60.7%).
• The texture bias in self-supervised models is not alleviated by contrastive learning objectives, suggesting that data augmentations play a key role in shaping this bias.
• Despite the fundamentally different training objective, self-supervised models do not show increased similarity to human visual perception in their behavior, indicating that current approaches still fall short of human-like invariance and robustness.