[Paper Review] Efficiency 360: Efficient Vision Transformers

Badri N. Patro, Vijay Srinivas Agneeswaran|arXiv (Cornell University)|Feb 16, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper introduces the Efficiency 360 framework, a holistic approach to optimizing Vision Transformers for industrial use by addressing memory efficiency, computational cost, accuracy, robustness, fairness, privacy, transparency, and inclusiveness. It benchmarks 20+ vision transformer models across parameters, FLOPs, and performance on ImageNet-1K, CIFAR, and Oxford datasets, revealing that larger models (e.g., ViT-H, CvT-24) and ImageNet-22K pretraining improve transfer learning, while long-range benchmarks like LRA show significant challenges for vision transformers in long-context scenarios.

ABSTRACT

Transformers are widely used for solving tasks in natural language processing, computer vision, speech, and music domains. In this paper, we talk about the efficiency of transformers in terms of memory (the number of parameters), computation cost (number of floating points operations), and performance of models, including accuracy, the robustness of the model, and fair \& bias-free features. We mainly discuss the vision transformer for the image classification task. Our contribution is to introduce an efficient 360 framework, which includes various aspects of the vision transformer, to make it more efficient for industrial applications. By considering those applications, we categorize them into multiple dimensions such as privacy, robustness, transparency, fairness, inclusiveness, continual learning, probabilistic models, approximation, computational complexity, and spectral complexity. We compare various vision transformer models based on their performance, the number of parameters, and the number of floating point operations (FLOPs) on multiple datasets.

Motivation & Objective

• To develop a unified framework for evaluating Vision Transformers across multiple industrial efficiency dimensions beyond just accuracy and FLOPs.
• To benchmark state-of-the-art Vision Transformers on standard vision datasets (ImageNet-1K, CIFAR, Oxford) in terms of parameters, FLOPs, and performance.
• To investigate the impact of model size and pretraining data (ImageNet-1K vs. ImageNet-22K) on transfer learning performance.
• To evaluate Vision Transformers on the Long Range Arena (LRA) benchmark, highlighting their limitations in long-context visual tasks.
• To identify open research directions in fairness, privacy, transparency, and multimodal applications of efficient transformers.

Proposed method

• Proposes the Efficiency 360 framework, a 360-degree evaluation system integrating 10 dimensions: robustness, privacy, fairness, transparency, inclusiveness, continual learning, probabilistic modeling, approximation, computational complexity, and spectral complexity.
• Compares 20+ Vision Transformer models based on architecture, attention type (global/local), position embedding, network structure (isotropic/pyramid), and extra labels (e.g., spectral gating, pooling).
• Employs standard benchmarks: ImageNet-1K for image classification, CIFAR-10/100 for small-scale evaluation, and Oxford-IIIT-Flowers/Pets for fine-grained recognition.
• Evaluates model performance on the Long Range Arena (LRA) benchmark, which tests long-context reasoning across diverse modalities including visual sequences of pixels.
• Uses Grad-CAM for model interpretability and visualization of attention-based explanations in DeiT-based inference.
• Analyzes architectural behavior across layers in ViT and CvT models to assess the effect of depth and patch size on performance.

Experimental results

Research questions

• RQ1 How do different Vision Transformer architectures compare in terms of parameters, FLOPs, and accuracy across standard vision benchmarks?
• RQ2 Does pretraining on ImageNet-22K consistently improve transfer learning performance compared to ImageNet-1K, and is this due to data scale or model capacity?
• RQ3 To what extent do Vision Transformers generalize to long-range visual sequences, as tested on the Long Range Arena (LRA) benchmark?
• RQ4 What are the key inefficiencies in current Vision Transformers when handling long sequences, and how do they compare to NLP-optimized efficient transformers?
• RQ5 How can the Efficiency 360 framework guide the design of future vision transformers with improved fairness, privacy, and transparency?

Key findings

• Larger Vision Transformer variants (e.g., ViT-H, CvT-24) outperform smaller counterparts (ViT-Base, CvT-13) on ImageNet-1K and fine-grained datasets, indicating that model scale improves performance.
• ViT-H with 16-patch size achieves better accuracy than ViT-Base and ViT-Large on ImageNet-1K, suggesting optimal patch size matters for performance.
• CvT-24 outperforms CvT-13 and CvT-21, reinforcing the trend that larger model capacity improves results across multiple datasets.
• Pretraining on ImageNet-22K does not consistently yield better transfer learning performance than ImageNet-1K, and the reason remains unclear—possibly due to model architecture or data distribution differences.
• Vision Transformers perform poorly on the Long Range Arena (LRA) benchmark, especially in visual tasks like Path-X and image sequence classification, indicating limited long-range modeling capability.
• No recent Vision Transformers (e.g., Swin, CSwin, RegionViT) have been evaluated on the LRA benchmark, highlighting a research gap in long-context visual reasoning.