[Paper Review] Vision Transformer: Vit and its Derivatives

Zujun Fu|arXiv (Cornell University)|May 12, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper reviews Vision Transformer (ViT) and its key derivatives, including PVT, Swin Transformer, MLP-Mixer, XCiT, MViT, TimesFormer, SegFormer, and UNETR, demonstrating how self-attention mechanisms and architectural innovations enable state-of-the-art performance across image classification, object detection, semantic segmentation, video recognition, and medical imaging. The core contribution lies in showing that ViT-based models achieve superior accuracy and generalization, especially with scaling, large-scale pretraining, and architectural modifications like local attention, efficient attention, and MLP-based mixing.

ABSTRACT

Transformer, an attention-based encoder-decoder architecture, has not only revolutionized the field of natural language processing (NLP), but has also done some pioneering work in the field of computer vision (CV). Compared to convolutional neural networks (CNNs), the Vision Transformer (ViT) relies on excellent modeling capabilities to achieve very good performance on several benchmarks such as ImageNet, COCO, and ADE20k. ViT is inspired by the self-attention mechanism in natural language processing, where word embeddings are replaced with patch embeddings. This paper reviews the derivatives of ViT and the cross-applications of ViT with other fields.

Motivation & Objective

• To survey the evolution and architectural innovations of Vision Transformers (ViT) and their derivatives across computer vision tasks.
• To analyze how modifications to self-attention mechanisms improve efficiency and performance on dense prediction and high-resolution tasks.
• To evaluate the impact of scaling model size and pretraining data on few-shot generalization and accuracy.
• To explore cross-field applications of ViT, including video recognition, semantic segmentation, and 3D medical image segmentation.
• To identify architectural principles—such as local attention, MLP-based mixing, and positional encoding removal—that enhance ViT performance

Proposed method

• Proposes Pyramid Vision Transformer (PVT) with Spatial-Reduced Attention (SRA) to reduce quadratic attention complexity by downsampling keys and values, enabling hierarchical feature learning.
• Introduces PVT-v2 with overlapping patch embedding, depthwise convolutional feedforward networks, and linear-complexity self-attention to improve local inductive bias and flexibility across image resolutions.
• Develops Swin Transformer using shifted window partitioning to enable local self-attention with linear complexity O(M×N), allowing hierarchical representation learning and global context modeling.
• Applies scaling laws to ViT by training a 2B-parameter model on 3B images, demonstrating improved few-shot accuracy (84.86% top-1 on ImageNet) and sample efficiency.
• Replaces self-attention with two MLPs in MLP-Mixer: one for channel mixing across patches and one for token mixing across spatial positions.
• Introduces XCiT with cross-covariance attention (XCA) that decouples token and channel mixing using depthwise and pointwise convolutions, enabling efficient, non-local attention with reduced parameters.

Experimental results

Research questions

• RQ1 How can self-attention mechanisms in ViT be made more efficient for high-resolution and dense prediction tasks?
• RQ2 What architectural modifications enable ViT to achieve performance comparable to CNNs in object detection and semantic segmentation?
• RQ3 To what extent does scaling model size and pretraining data improve few-shot generalization in ViT-based models?
• RQ4 Can self-attention be effectively replaced by MLP-based or convolution-based mixing mechanisms without sacrificing performance?
• RQ5 How do architectural choices like positional encoding, pooling, and normalization affect ViT’s robustness across varying input resolutions?

Key findings

• Swin Transformer achieves linear complexity O(M×N) by using shifted window partitioning, enabling efficient local self-attention and hierarchical feature learning.
• PVT-v2 improves local inductive bias via overlapping patches and depthwise convolution, achieving better performance on semantic segmentation and object detection.
• Scaling ViT to 2 billion parameters with 3 billion images yields 90.45% top-1 accuracy on ImageNet and 84.86% few-shot accuracy, demonstrating strong generalization from limited data.
• Decoupling weight decay for the head and body layers improves few-shot performance, with stronger decay on the head yielding better class separation.
• MLP-Mixer and XCiT achieve competitive performance by replacing self-attention with two-stage MLP or convolution-based mixing, reducing computational cost while maintaining accuracy.
• SegFormer achieves state-of-the-art results in semantic segmentation by using a hierarchical encoder without positional encoding and a simple MLP decoder, enabling robustness to resolution changes.