[Paper Review] Do Vision Transformers See Like Convolutional Neural Networks?

Maithra Raghu, Thomas Unterthiner+3 more|arXiv (Cornell University)|Aug 19, 2021

Domain Adaptation and Few-Shot Learning50 references109 citations

TL;DR

The paper analyzes Vision Transformers (ViTs) and CNNs to understand how ViTs represent visual information, revealing key differences in representation structure, attention behavior, skip connections, spatial localization, and the impact of data scale on transfer learning.

ABSTRACT

Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two architectures, such as ViT having more uniform representations across all layers. We explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connections, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification methods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.

Motivation & Objective

• Understand how ViTs represent visual information compared to CNNs.
• Characterize the internal representation structure across ViTs and CNNs.
• Investigate how self-attention and skip connections shape information propagation.
• Examine how ViTs preserve spatial information and localization.
• Assess the impact of dataset scale on intermediate representations and transfer learning.

Proposed method

• Use Centered Kernel Alignment (CKA) to quantify representation similarity across layers within and between ViTs and CNNs.
• Compare ViT and CNN architectures (ViT-B/32, ViT-B/16, ViT-L/16, ViT-H/14 vs. ResNet-50x1, ResNet-152x2) trained on JFT-300M or ImageNet.
• Analyze attention distances in ViT self-attention heads to assess local vs global information aggregation.
• Intervene by removing skip connections to test their role in representation propagation.
• Evaluate spatial localization by comparing final-layer token representations to input patches via CKA, and use linear probes to assess localization capability.
• Study the effect of pretraining data scale on intermediate representations using linear probes across model sizes.

Experimental results

Research questions

• RQ1 Do ViTs develop representation structures that are more uniform across layers compared to CNNs?
• RQ2 How do self-attention and skip connections influence information propagation and feature representations in ViTs vs CNNs?
• RQ3 Do ViTs preserve input spatial information in higher layers, and how does this affect localization tasks?
• RQ4 How does pretraining data scale affect the quality of intermediate representations and transfer learning for ViTs?
• RQ5 What are the implications of ViTs’ representations for future architectures and tasks such as object detection?

Key findings

• ViTs exhibit highly uniform representations across layers, unlike CNNs which show clear stage-based similarity patterns.
• Lower ViT layers access both local and global information, while higher ViT layers rely on global information, unlike CNNs with fixed local receptive fields.
• Skip connections in ViTs strongly propagate representations across layers, with a phase transition from CLS token to spatial token propagation in deeper layers.
• ViTs preserve input spatial information in higher layers, with localization influenced by CLS-based classification; GAP pooling reduces localization.
• Dataset scale (e.g., JFT-300M) significantly improves high-level/intermediate representations, especially for larger ViT models.
• Lower layers of ViTs can resemble ResNet features with local heads, but globally-attentive heads compute qualitatively different representations.