[Paper Review] A Survey on Visual Transformer

Kai Han, Yunhe Wang+11 more|arXiv (Cornell University)|Dec 23, 2020

Advanced Neural Network Applications70 references220 citations

TL;DR

This survey reviews vision transformer models across backbone learning, high/mid-level vision, low-level vision, and video tasks, analyzing advantages, limitations, and efficient variants.

ABSTRACT

Transformer, first applied to the field of natural language processing, is a type of deep neural network mainly based on the self-attention mechanism. Thanks to its strong representation capabilities, researchers are looking at ways to apply transformer to computer vision tasks. In a variety of visual benchmarks, transformer-based models perform similar to or better than other types of networks such as convolutional and recurrent neural networks. Given its high performance and less need for vision-specific inductive bias, transformer is receiving more and more attention from the computer vision community. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages. The main categories we explore include the backbone network, high/mid-level vision, low-level vision, and video processing. We also include efficient transformer methods for pushing transformer into real device-based applications. Furthermore, we also take a brief look at the self-attention mechanism in computer vision, as it is the base component in transformer. Toward the end of this paper, we discuss the challenges and provide several further research directions for vision transformers.

Motivation & Objective

• Survey the development of vision transformer models categorized by application (backbone, high/mid-level, low-level, video).
• Analyze core components (self-attention, positional encoding, architecture variants) and efficiency methods for real-device deployment.
• Discuss challenges, trade-offs, and potential research directions in vision transformers.
• Provide comparisons of representative models and summarize key findings to guide future research.

Proposed method

• Explain standard transformer components and self-attention equations (Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V).
• Describe ViT and variants that adapt transformers for images (patch embeddings, positional encodings, class token).
• Survey backbone, high/mid-level vision, low-level vision, and video processing models and efficiency approaches.
• Summarize self-supervised and generative/self-supervised pretraining methods (iGPT, MAE, SimMIM) and contrastive learning (MoCo v3).
• Compare CNN+Transformer hybrids and pure transformer backbones with quantitative results where available.

Experimental results

Research questions

• RQ1 What are the key categories and tasks where vision transformers have been applied?
• RQ2 What are the main architectural variants and techniques to improve locality, efficiency, and performance in vision transformers?
• RQ3 How do vision transformers compare to CNNs in terms of accuracy, throughput, and data efficiency across tasks?
• RQ4 What are the effective pretraining strategies (supervised, self-supervised, generative) for vision transformers?
• RQ5 What are the open challenges and future directions for vision transformers?

Key findings

• Vision transformers provide competitive or superior performance on many visual benchmarks, approaching or exceeding CNNs in several settings.
• Pure transformer backbones like ViT require large-scale pretraining to surpass CNNs, with data efficiency improved via DeiT and distillation.
• Locality-enhancing variants (TNT, Swin, RegionViT, etc.) and convolution-integrated hybrids (CvT, CvViT, LeViT) improve data efficiency and real-time performance.
• Self-supervised and generative pretraining (iGPT, MAE, SimMIM, MoCo v3) enable strong representations with less labeled data.
• Efficient transformer designs (windowed attention, hierarchical pyramids, NAS-inspired architectures) balance accuracy, FLOPs, and throughput for practical deployment.
• In benchmark comparisons, diverse models (e.g., DeiT, Swin, TNT, ViT variants) achieve top-1 accuracies in the 79–86% range on ImageNet with varying parameter counts and compute.