[Paper Review] Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition

Yulin Wang, Rui Huang+3 more|arXiv (Cornell University)|May 31, 2021

Advanced Neural Network Applications70 citations

TL;DR

The paper introduces Dynamic Vision Transformer (DVT), which adaptively selects the number of tokens per input by cascading transformers and using early exit, plus feature and relationship reuse to improve efficiency without sacrificing accuracy.

ABSTRACT

Vision Transformers (ViT) have achieved remarkable success in large-scale image recognition. They split every 2D image into a fixed number of patches, each of which is treated as a token. Generally, representing an image with more tokens would lead to higher prediction accuracy, while it also results in drastically increased computational cost. To achieve a decent trade-off between accuracy and speed, the number of tokens is empirically set to 16x16 or 14x14. In this paper, we argue that every image has its own characteristics, and ideally the token number should be conditioned on each individual input. In fact, we have observed that there exist a considerable number of "easy" images which can be accurately predicted with a mere number of 4x4 tokens, while only a small fraction of "hard" ones need a finer representation. Inspired by this phenomenon, we propose a Dynamic Transformer to automatically configure a proper number of tokens for each input image. This is achieved by cascading multiple Transformers with increasing numbers of tokens, which are sequentially activated in an adaptive fashion at test time, i.e., the inference is terminated once a sufficiently confident prediction is produced. We further design efficient feature reuse and relationship reuse mechanisms across different components of the Dynamic Transformer to reduce redundant computations. Extensive empirical results on ImageNet, CIFAR-10, and CIFAR-100 demonstrate that our method significantly outperforms the competitive baselines in terms of both theoretical computational efficiency and practical inference speed. Code and pre-trained models (based on PyTorch and MindSpore) are available at https://github.com/blackfeather-wang/Dynamic-Vision-Transformer and https://github.com/blackfeather-wang/Dynamic-Vision-Transformer-MindSpore.

Motivation & Objective

• Motivate efficient image recognition by showing fixed-token ViTs waste computation on easy images.
• Propose a dynamic token-configuration framework that adapts per input to balance accuracy and efficiency.
• Develop mechanisms for reusing features and self-attention relationships to reduce redundant computation.
• Demonstrate empirical efficiency and accuracy gains across standard benchmarks and backbones.

Proposed method

• Cascade multiple Transformers with increasing token counts (e.g., 7x7, 10x10, 14x14).
• Early-termination: stop inference when prediction confidence exceeds learned thresholds.
• Train with a joint objective that sums cross-entropy losses from all exits.
• Feature reuse: inject context embeddings from upstream final features into downstream MLPs.
• Relationship reuse: transfer and adapt upstream self-attention logits to refine downstream attention maps (A_l + r_l(A_up)).
• Adapt bottleneck forward passes to achieve online adjustable FLOPs while maintaining accuracy.

Experimental results

Research questions

• RQ1 Can per-input token budgeting improve ViT efficiency without sacrificing accuracy on standard benchmarks?
• RQ2 How can we reuse features and attention across cascaded transformers to avoid redundant computation?
• RQ3 What is the impact of adaptive token counts and early exits on real-world inference speed and FLOPs?
• RQ4 How do DVT variants perform across different backbones (e.g., T2T-ViT, DeiT) and datasets (ImageNet, CIFAR-10/100)?

Key findings

• DVT consistently reduces backbones’ computational cost while preserving or improving accuracy on ImageNet.
• On ImageNet, DVT achieves higher accuracy with substantially fewer FLOPs compared to vanilla backbones (e.g., up to 3.6x fewer FLOPs for similar accuracy).
• Two reuse mechanisms (feature reuse and relationship reuse) significantly boost exit accuracies with modest extra computation.
• DVT yields practical speedups on GPUs (e.g., improved throughput and real inference speed) and outperforms several state-of-the-art efficiency methods at similar FLOPs.
• Ablation shows both reuse mechanisms contribute to performance, with cumulative gains when used together.