[Paper Review] A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking

Lorenzo Papa, Paolo Russo+2 more|arXiv (Cornell University)|Sep 5, 2023

CCD and CMOS Imaging Sensors81 references4 citations

TL;DR

This survey proposes a comprehensive analysis of four key efficiency strategies—compact architecture, pruning, knowledge distillation, and quantization—for Vision Transformers (ViTs), introducing a novel metric, Efficient Error Rate (EER), to fairly benchmark model efficiency. It identifies that hybrid strategies combining multiple techniques, such as DynamicViT’s integration of pruning and knowledge distillation, offer the best trade-off between performance and computational cost.

ABSTRACT

Vision Transformer (ViT) architectures are becoming increasingly popular and widely employed to tackle computer vision applications. Their main feature is the capacity to extract global information through the self-attention mechanism, outperforming earlier convolutional neural networks. However, ViT deployment and performance have grown steadily with their size, number of trainable parameters, and operations. Furthermore, self-attention's computational and memory cost quadratically increases with the image resolution. Generally speaking, it is challenging to employ these architectures in real-world applications due to many hardware and environmental restrictions, such as processing and computational capabilities. Therefore, this survey investigates the most efficient methodologies to ensure sub-optimal estimation performances. More in detail, four efficient categories will be analyzed: compact architecture, pruning, knowledge distillation, and quantization strategies. Moreover, a new metric called Efficient Error Rate has been introduced in order to normalize and compare models' features that affect hardware devices at inference time, such as the number of parameters, bits, FLOPs, and model size. Summarizing, this paper firstly mathematically defines the strategies used to make Vision Transformer efficient, describes and discusses state-of-the-art methodologies, and analyzes their performances over different application scenarios. Toward the end of this paper, we also discuss open challenges and promising research directions.

Motivation & Objective

• To systematically analyze and compare four major efficiency strategies—compact architecture, pruning, knowledge distillation, and quantization—for Vision Transformers.
• To address the challenge of deploying computationally heavy ViTs on resource-constrained devices due to quadratic growth in self-attention computation with image resolution.
• To introduce a new evaluation metric, Efficient Error Rate (EER), to normalize and compare model efficiency across hardware constraints such as FLOPs, parameters, and model size.
• To identify the optimal balance between model accuracy and inference efficiency for general-purpose computer vision applications.
• To highlight open challenges in generalization, evaluation metrics, and multi-strategy fusion for future research.

Proposed method

• Categorize and formalize four main efficiency techniques: compact architecture (CA), pruning (P), knowledge distillation (KD), and quantization (Q), with mathematical formulations of their core mechanisms.
• Introduce the Efficient Error Rate (EER) metric, which normalizes performance across models by comparing their error rates to a non-efficient baseline while accounting for FLOPs, parameters, bits, and model size.
• Review state-of-the-art methods in each category, analyzing their strengths, weaknesses, and architectural innovations.
• Benchmark models across standard datasets (e.g., ImageNet) using standardized metrics including Top-1 accuracy, FLOPs, and parameter count.
• Evaluate model performance across diverse application scenarios, including classification, segmentation, and depth estimation.
• Propose a multi-strategy fusion framework, exemplified by DynamicViT, to combine techniques like pruning and knowledge distillation to minimize accuracy loss while maximizing efficiency.

Figure 1: Graphical overview of the vanilla self-attention and the multi-head self-attention blocks. The $\mathcal{O}(n^{2})$ in the softmax dot-product self-attention highlights the quadratic cost of each operation.

Experimental results

Research questions

• RQ1 Which of the four efficiency strategies—compact architecture, pruning, knowledge distillation, or quantization—delivers the best trade-off between model accuracy and computational efficiency?
• RQ2 How can a unified metric fairly compare models with varying hardware footprints and performance characteristics?
• RQ3 To what extent do hybrid strategies that combine multiple efficiency techniques outperform single-method approaches?
• RQ4 What are the limitations of current evaluation protocols in capturing real-world deployment constraints such as latency and memory usage?
• RQ5 How do efficient ViTs perform under adversarial conditions or on non-classification tasks, and can they generalize across diverse vision tasks?

Key findings

• The Efficient Error Rate (EER) metric successfully normalizes and enables fair comparison of ViT models across different hardware constraints, offering a more holistic view of efficiency than traditional metrics.
• No single strategy excels across all metrics; however, hybrid approaches like DynamicViT, which combine pruning and knowledge distillation, achieve superior performance with minimal accuracy degradation.
• Knowledge distillation and quantization are particularly effective in reducing model size and inference FLOPs without significant accuracy loss, especially when combined with architectural compression.
• Pruning techniques, especially structured pruning, significantly reduce parameters and FLOPs while maintaining high Top-1 accuracy, especially when applied post-training.
• Compact architecture designs, such as Swin Transformer and MobileViT, demonstrate strong performance on low-resource devices due to their efficient self-attention mechanisms and hierarchical feature extraction.
• The study identifies that current benchmarks are largely limited to image classification, and future work must expand evaluation to dense prediction tasks and adversarial robustness.

Figure 2: Graphical representation of the vanilla KD learning strategy. Please refer to Section 2.3 for the used notation.