[Paper Review] Partial success in closing the gap between human and machine vision

Robert Geirhos, Kantharaju Narayanappa+5 more|arXiv (Cornell University)|Jun 14, 2021

Domain Adaptation and Few-Shot Learning94 references63 citations

TL;DR

The study shows that robustness to out-of-distribution distortions is increasingly matched or exceeded by modern models, especially data-rich and transformer-based ones, but an image-level error pattern gap between humans and machines persists. A large-scale psychophysical benchmark with 17 OOD datasets and 85,120 trials evaluates various model families to quantify progress toward human-like vision.

ABSTRACT

A few years ago, the first CNN surpassed human performance on ImageNet. However, it soon became clear that machines lack robustness on more challenging test cases, a major obstacle towards deploying machines "in the wild" and towards obtaining better computational models of human visual perception. Here we ask: Are we making progress in closing the gap between human and machine vision? To answer this question, we tested human observers on a broad range of out-of-distribution (OOD) datasets, recording 85,120 psychophysical trials across 90 participants. We then investigated a range of promising machine learning developments that crucially deviate from standard supervised CNNs along three axes: objective function (self-supervised, adversarially trained, CLIP language-image training), architecture (e.g. vision transformers), and dataset size (ranging from 1M to 1B). Our findings are threefold. (1.) The longstanding distortion robustness gap between humans and CNNs is closing, with the best models now exceeding human feedforward performance on most of the investigated OOD datasets. (2.) There is still a substantial image-level consistency gap, meaning that humans make different errors than models. In contrast, most models systematically agree in their categorisation errors, even substantially different ones like contrastive self-supervised vs. standard supervised models. (3.) In many cases, human-to-model consistency improves when training dataset size is increased by one to three orders of magnitude. Our results give reason for cautious optimism: While there is still much room for improvement, the behavioural difference between human and machine vision is narrowing. In order to measure future progress, 17 OOD datasets with image-level human behavioural data and evaluation code are provided as a toolbox and benchmark at: https://github.com/bethgelab/model-vs-human/

Motivation & Objective

• Assess whether the robustness gap between human and machine vision is narrowing on out-of-distribution data.
• Evaluate how different ML developments (objective functions, architectures, data size) affect human–machine alignment.
• Provide a benchmark toolbox and datasets to track future progress in this area.

Proposed method

• Collected 85,120 psychophysical trials from 90 human observers on 17 OOD datasets designed to test distortion robustness.
• Compared 52 models spanning CNNs, self-supervised, adversarially trained, vision transformers, and large-data/noisy-label regimes.
• Evaluated models using OOD accuracy and three alignment metrics: Accuracy difference A(m), Observed consistency O(m), and Error consistency E(m).
• Opened the model-vs-human toolbox to benchmark new models against human data.
• Mapped ImageNet 1000 classes to 16 categories for human-model comparability using WordNet hierarchy.

Experimental results

Research questions

• RQ1 Do modern ML models close the distortion robustness gap with humans on a broad set of OOD conditions?
• RQ2 How do objective functions, architectures, and training data scale influence human–machine alignment across images?
• RQ3 To what extent do machines and humans share or differ in error patterns on individual images under OOD conditions?

Key findings

• Best models trained on large-scale data match or exceed human feedforward accuracy on most OOD datasets.
• There remains a substantial image-level consistency gap: models and humans often err on different images, though data-rich models can narrow this gap on some datasets.
• Self-supervised models show limited robustness gains over supervised baselines, with notable improvements largely attributable to data augmentation choices.
• Adversarially trained models increase robustness yet can become more vulnerable to non-adversarial perturbations and exhibit stronger texture bias.
• Vision transformers and large-scale data substantially improve OOD performance, with CLIP achieving near-human error patterns on some metrics.
• The paper provides a toolbox and 17 OOD datasets to benchmark future progress and quantify human–machine behavioral alignment.