[Paper Review] Are Convolutional Neural Networks or Transformers more like human vision?

Shikhar Tuli, Ishita Dasgupta+2 more|arXiv (Cornell University)|May 15, 2021

Visual Attention and Saliency Detection99 citations

TL;DR

The paper compares CNNs and Vision Transformers (ViT) on a human-vision similarity basis using error-consistency metrics, showing ViT aligns more with human errors and shape bias on Stylized ImageNet, with nuances after fine-tuning.

ABSTRACT

Modern machine learning models for computer vision exceed humans in accuracy on specific visual recognition tasks, notably on datasets like ImageNet. However, high accuracy can be achieved in many ways. The particular decision function found by a machine learning system is determined not only by the data to which the system is exposed, but also the inductive biases of the model, which are typically harder to characterize. In this work, we follow a recent trend of in-depth behavioral analyses of neural network models that go beyond accuracy as an evaluation metric by looking at patterns of errors. Our focus is on comparing a suite of standard Convolutional Neural Networks (CNNs) and a recently-proposed attention-based network, the Vision Transformer (ViT), which relaxes the translation-invariance constraint of CNNs and therefore represents a model with a weaker set of inductive biases. Attention-based networks have previously been shown to achieve higher accuracy than CNNs on vision tasks, and we demonstrate, using new metrics for examining error consistency with more granularity, that their errors are also more consistent with those of humans. These results have implications both for building more human-like vision models, as well as for understanding visual object recognition in humans.

Motivation & Objective

• Motivate beyond-accuracy analyses by examining error patterns to assess human-likeness of models.
• Compare CNNs and Vision Transformers on error consistency with human classification behavior.
• Introduce granular metrics to analyze misclassifications beyond overall accuracy.
• Investigate how data augmentation and fine-tuning affect shape bias and human-like behavior.

Proposed method

• Compare ResNet (CNN) and ViT (Vision Transformer) models on Stylized ImageNet (SIN), a diagnostic dataset with texture-shape cue conflicts.
• Use error-consistency measures including observed error overlap, Cohen’s kappa correction, and Jensen-Shannon (JS) distances on confusion matrices.
• Construct class-wise and inter-class JS distances from 16 entry-level categories and 240 off-diagonal error types.
• Analyze shape bias by evaluating SIN trials where shape vs. texture predicts the true label.
• Fine-tune ViT and ResNet with augmented data to study shifts in error-consistency and shape bias.

Experimental results

Research questions

• RQ1 Do Vision Transformers produce error patterns more consistent with humans than CNNs on SIN?
• RQ2 How do granular error-mariance metrics (class-wise and inter-class JS distances) compare to Cohen’s kappa in reflecting human-like behavior?
• RQ3 What is the relationship between shape bias and error-consistency with humans across architectures?
• RQ4 How does fine-tuning with augmented data influence shape bias and human-likeness of CNNs and ViTs?

Key findings

• ViT is more consistent with humans than ResNet in terms of error patterns on SIN using Cohen’s kappa and class-wise JS distance.
• Class-wise JS distance aligns with Cohen’s kappa and shows ViT’s human-like error patterns, independent of SIN training accuracy.
• Inter-class JS distance reveals that ViT has a higher degree of error diversity (more human-like in some respects) than ResNet when considering full error distributions.
• ViT exhibits higher shape bias than CNNs on SIN, partially explaining higher shape-based error consistency.
• Fine-tuning with augmented data increases shape bias in both ResNet and ViT; accuracy on ImageNet can decrease slightly, with ResNet more affected.