[Paper Review] Attention mechanisms and deep learning for machine vision: A survey of the state of the art

Abdul Mueed Hafiz, Shabir A. Parah+1 more|arXiv (Cornell University)|Jun 3, 2021

Advanced Neural Network Applications5 citations

TL;DR

This survey provides a comprehensive overview of attention mechanisms and deep learning in machine vision, focusing on vision transformers (ViTs) and their hybridization with convolutional neural networks (CNNs). It analyzes self-attention, masked attention, and multi-head attention mechanisms, evaluates ViT performance across tasks like image classification and object detection, and identifies key challenges such as data hunger and high computational cost, while advocating for hybrid CNN-ViT architectures to balance efficiency and accuracy.

ABSTRACT

With the advent of state of the art nature-inspired pure attention based models i.e. transformers, and their success in natural language processing (NLP), their extension to machine vision (MV) tasks was inevitable and much felt. Subsequently, vision transformers (ViTs) were introduced which are giving quite a challenge to the established deep learning based machine vision techniques. However, pure attention based models/architectures like transformers require huge data, large training times and large computational resources. Some recent works suggest that combinations of these two varied fields can prove to build systems which have the advantages of both these fields. Accordingly, this state of the art survey paper is introduced which hopefully will help readers get useful information about this interesting and potential research area. A gentle introduction to attention mechanisms is given, followed by a discussion of the popular attention based deep architectures. Subsequently, the major categories of the intersection of attention mechanisms and deep learning for machine vision (MV) based are discussed. Afterwards, the major algorithms, issues and trends within the scope of the paper are discussed.

Motivation & Objective

• To provide a systematic review of attention mechanisms and their integration with deep learning in machine vision.
• To analyze the strengths and limitations of vision transformers (ViTs) compared to traditional CNN-based models.
• To explore hybrid architectures that combine CNNs and transformers to leverage the advantages of both approaches.
• To identify open research challenges in data efficiency, training time, and computational resource requirements for ViTs.
• To guide researchers and practitioners toward effective use of attention-based models in machine vision applications.

Proposed method

• Surveying 110+ research papers to analyze attention mechanisms and their application in deep learning for machine vision.
• Introducing self-attention as a mechanism that computes relevance scores between all elements in a sequence using queries, keys, and values.
• Explaining masked self-attention to prevent information leakage in autoregressive modeling, using a triangular mask matrix.
• Describing multi-head attention as a mechanism that enables modeling of diverse dependencies by applying multiple attention heads in parallel.
• Reviewing three main categories of hybrid models: attention-augmented CNNs (e.g., CBAM), CNN-teacher/transformer-student pipelines, and end-to-end CNN-transformer architectures.
• Evaluating the role of network architecture search (NAS) in optimizing hybrid CNN-transformer search spaces, though noting its computational cost.

Experimental results

Research questions

• RQ1 How do self-attention mechanisms enable global feature modeling in vision transformers compared to local receptive fields in CNNs?
• RQ2 What are the key performance and efficiency trade-offs between pure vision transformers and CNN-based models in machine vision tasks?
• RQ3 In what ways can hybrid architectures combining CNNs and transformers improve performance while reducing data and computational demands?
• RQ4 What are the major open challenges in training vision transformers, such as data hunger and long training times?
• RQ5 How can attention mechanisms be effectively integrated into existing deep learning pipelines for improved machine vision system design?

Key findings

• Vision transformers (ViTs) achieve performance on par with or better than CNNs in image classification, object detection, segmentation, and video understanding tasks.
• ViTs are highly data-hungry, requiring pre-training on large-scale datasets like JFT-300M before fine-tuning on ImageNet-1K to achieve strong results.
• Despite their performance, ViTs require significantly longer training times and high computational resources, making them less practical for resource-constrained environments.
• Hybrid models that integrate attention mechanisms into CNNs (e.g., CBAM) show performance gains with minimal computational overhead, indicating strong potential for incremental improvement.
• Two-stage pipelines using CNNs as feature extractors and transformers as classifiers show dependency on the quality of the CNN’s feature representation, highlighting the importance of backbone design.
• End-to-end hybrid architectures combining CNNs and transformers via NAS are promising but remain computationally expensive and require further optimization for broader deployment.