[论文解读] Transformers Meet Visual Learning Understanding: A Comprehensive Review

Yuting Yang, Licheng Jiao+5 more|arXiv (Cornell University)|Mar 24, 2022

Advanced Neural Network Applications被引用 21

一句话总结

A comprehensive survey of Transformer-based models in image and video understanding, detailing attention mechanisms, visual Transformer modules, backbone/neck designs, and performance trends across image classification, detection, segmentation, tracking, and video classification.*

ABSTRACT

Dynamic attention mechanism and global modeling ability make Transformer show strong feature learning ability. In recent years, Transformer has become comparable to CNNs methods in computer vision. This review mainly investigates the current research progress of Transformer in image and video applications, which makes a comprehensive overview of Transformer in visual learning understanding. First, the attention mechanism is reviewed, which plays an essential part in Transformer. And then, the visual Transformer model and the principle of each module are introduced. Thirdly, the existing Transformer-based models are investigated, and their performance is compared in visual learning understanding applications. Three image tasks and two video tasks of computer vision are investigated. The former mainly includes image classification, object detection, and image segmentation. The latter contains object tracking and video classification. It is significant for comparing different models' performance in various tasks on several public benchmark data sets. Finally, ten general problems are summarized, and the developing prospects of the visual Transformer are given in this review.

研究动机与目标

• Assess how attention mechanisms underpin Transformer performance in visual learning tasks.
• Summarize core visual Transformer architectures and module designs (backbones, encoders/decoders, position encoding).
• Survey Transformer-based methods for image classification, object detection, segmentation, tracking, and video classification.
• Compare performance on public benchmarks and outline current challenges and future directions.

提出的方法

• Review the spectrum of attention mechanisms (channel, spatial, temporal, branch) and their roles in Transformers.
• Explain the architecture of core visual Transformer modules (self-attention, multi-head attention, FFN, position encoding) and their computational complexity.
• Catalog Transformer backbones (e.g., Swin, CSWin, PVT, CrossFormer) and discuss their design principles (hierarchy, windowed/local attention, cross-scale).
• Summarize Transformer-based methods for image tasks (classification, detection, segmentation) and video tasks (tracking, video classification) with performance comparisons on public datasets.
• Highlight notable pre-training strategies (ViT, iGPT, DeiT, CrossViT, etc.) and their impact on data efficiency and accuracy.

实验结果

研究问题

• RQ1 What are the main attention mechanisms used in visual Transformers and how do they influence performance across image and video tasks?
• RQ2 How have Transformer backbones and module designs evolved for image classification, detection, and segmentation?
• RQ3 What are the current performance trends and benchmark outcomes for Transformer-based visual learning methods on standard datasets?
• RQ4 What ten public challenges or open issues face Transformer-based visual learning and how might they be addressed?
• RQ5 What pre-training strategies enable data-efficient and high-accuracy Transformer models in vision?

主要发现

• Transformer-based methods have achieved state-of-the-art or competitive results across image classification, detection, segmentation, tracking, and video classification.
• Notable backbones (Swin, CSWin, PVT, CrossFormer) introduce hierarchical, windowed, or cross-scale attention to balance accuracy and computational cost.
• Pre-training strategies (ViT, iGPT, DeiT, CrossViT, etc.) significantly impact data efficiency and downstream performance as shown on benchmarks like ImageNet and COCO.
• The review provides cross-task performance comparisons on public benchmarks to facilitate experimental choices for researchers.
• The authors summarize ten general challenges and offer directions for future Transformer research in visual learning understanding.