[Paper Review] A Survey of Visual Transformers

Yang Liu, Yao Zhang+8 more|arXiv (Cornell University)|Nov 11, 2021

Advanced Neural Network ApplicationsComputer Science134 references42 citations

TL;DR

The paper comprehensively surveys 100+ visual Transformer models for classification, detection, and segmentation across image, point cloud, and multi-sensory data, providing taxonomy, comparisons, and future directions.

ABSTRACT

Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern Convolution Neural Networks (CNNs). In this survey, we have reviewed over one hundred of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where a taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, three promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.

Motivation & Objective

• Provide a systematic overview of visual Transformer architectures across CV tasks (classification, detection, segmentation).
• Organize existing methods by data streams (images, point clouds, multi-sensory data) and motivations.
• Analyze architectural components and training schemes to reveal strengths, limitations, and connections to CNNs.
• Suggest future research directions and practical considerations for leveraging visual Transformers in diverse tasks.

Proposed method

• Review and categorize over 100 visual Transformer works according to three CV tasks and data stream types.
• Propose a taxonomy of backbones and components (e.g., patch-based, query-based, hierarchical, local/global attention).
• Summarize training strategies, including self-supervision and pre-training data considerations.
• Compare methods under varying configurations and data regimes to provide intuitive, cross-study insights.
• Discuss unexploited aspects and potential bridging mechanisms between visual Transformers and sequential models.

Experimental results

Research questions

• RQ1 What are the main architectural families of visual Transformers used for CV tasks?
• RQ2 How do visual Transformers perform across image classification, detection, and segmentation, and what data streams do they operate on?
• RQ3 What training strategies and inductive biases help visual Transformers generalize with limited data?
• RQ4 What future directions and gaps exist for unified, multi-task, and multi-modal visual Transformers?

Key findings

• Visual Transformers have achieved impressive performance across classification, detection, and segmentation benchmarks compared to CNNs under various configurations.
• A spectrum of architectural trends exists (e.g., local vs. global attention, hierarchical designs, and deep transformer variants) with each offering trade-offs in accuracy and efficiency.
• Hybrid approaches that integrate CNN inductive biases (through convolutional stems, bias, or hybrid backbones) improve data efficiency and performance.
• Advances include local attention schemes (e.g., Swin), hierarchical tokens, and deep-transformer techniques to mitigate over-smoothing and attention collapse.
• Self-supervised and multi-modal pre-training strategies play a key role in enabling ViTs to perform well, often reducing the need for large external datasets.
• The paper highlights promising directions, such as encoder–decoder unification via query embeddings and potential multi-task, multi-stream fusion architectures.