[Paper Review] Intriguing Properties of Vision Transformers

Muzammal Naseer, Kanchana Ranasinghe+4 more|arXiv (Cornell University)|May 21, 2021

Advanced Neural Network ApplicationsComputer Science46 references301 citations

TL;DR

The paper demonstrates that Vision Transformers (ViTs) are highly robust to occlusions, domain shifts, and permutations, possess strong shape-bias relative to CNNs, can perform segmentation without pixel supervision via shape tokens, and provide transferable off-the-shelf features, all due to their flexible self-attention-based receptive fields.

ABSTRACT

Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. These models are based on multi-head self-attention mechanisms that can flexibly attend to a sequence of image patches to encode contextual cues. An important question is how such flexibility in attending image-wide context conditioned on a given patch can facilitate handling nuisances in natural images e.g., severe occlusions, domain shifts, spatial permutations, adversarial and natural perturbations. We systematically study this question via an extensive set of experiments encompassing three ViT families and comparisons with a high-performing convolutional neural network (CNN). We show and analyze the following intriguing properties of ViT: (a) Transformers are highly robust to severe occlusions, perturbations and domain shifts, e.g., retain as high as 60% top-1 accuracy on ImageNet even after randomly occluding 80% of the image content. (b) The robust performance to occlusions is not due to a bias towards local textures, and ViTs are significantly less biased towards textures compared to CNNs. When properly trained to encode shape-based features, ViTs demonstrate shape recognition capability comparable to that of human visual system, previously unmatched in the literature. (c) Using ViTs to encode shape representation leads to an interesting consequence of accurate semantic segmentation without pixel-level supervision. (d) Off-the-shelf features from a single ViT model can be combined to create a feature ensemble, leading to high accuracy rates across a range of classification datasets in both traditional and few-shot learning paradigms. We show effective features of ViTs are due to flexible and dynamic receptive fields possible via the self-attention mechanism.

Motivation & Objective

• Motivate investigation into robustness and generalization of ViTs vs CNNs under nuisances such as occlusions, domain shifts, and perturbations.
• Characterize how self-attention and dynamic receptive fields contribute to robustness and representation quality.
• Explore shape vs. texture biases in ViTs and their impact on performance and segmentation capabilities.
• Propose architectural ideas (shape token) to encode shape information within ViTs and assess transferability of off-the-shelf ViT features.
• Demonstrate practical benefits of ViTs for few-shot and diverse downstream tasks using single-model feature ensembles.

Proposed method

• Conduct extensive experiments across ViT family variants (ViT, DeiT, T2T) on fifteen vision datasets.
• Apply occlusion scenarios using PatchDrop with Random, Salient, and Non-salient masks to test robustness.
• Analyze texture vs. shape bias by training on stylized ImageNet and introducing a dedicated shape token.
• Investigate the role of positional encoding by shuffling image patches and varying patch granularity.
• Evaluate adversarial and common corruptions robustness, with and without augmentations.
• Demonstrate off-the-shelf ViT feature transfer by ensembling class tokens from different blocks and testing on downstream classification and few-shot tasks.

Experimental results

Research questions

• RQ1 Do Vision Transformers remain robust under severe occlusions and patch-level perturbations compared to CNNs?
• RQ2 Are ViTs biased toward texture or shape, and can shape-based cues be encoded without sacrificing accuracy?
• RQ3 Can ViTs achieve automated segmentation using shape information without pixel-level supervision?
• RQ4 Do off-the-shelf ViT features transfer better than CNN features across diverse downstream tasks and settings?
• RQ5 What is the role of positional encoding and receptive fields in ViT robustness and generalization?

Key findings

• ViTs retain high accuracy under heavy occlusion, e.g., DeiT can reach ~60% top-1 on ImageNet with 80% randomly occluded content, while CNNs collapse to near 0%.
• ViTs exhibit less texture bias and stronger shape-bias than CNNs, achieving shape recognition comparable to humans when trained to emphasize shapes.
• Shape-focused training and shape tokens enable ViTs to perform automated foreground segmentation without pixel-level supervision.
• Off-the-shelf ViT features transfer well to diverse datasets and enable strong ensembles from a single model, often outperforming CNN baselines in transfer tasks and few-shot learning.
• ViTs show permutation invariance to patch ordering and rely on dynamic, content-dependent receptive fields rather than positional encodings for robustness.