[Paper Review] Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length.

Yulin Wang, Rui Huang+3 more|arXiv (Cornell University)|May 31, 2021

Advanced Neural Network Applications42 references19 citations

TL;DR

This paper proposes a Dynamic Vision Transformer that adaptively adjusts the number of image tokens per input based on instance difficulty, using a cascade of progressively finer Transformers that terminate early upon confident predictions. The method achieves state-of-the-art efficiency and accuracy on ImageNet, CIFAR-10, and CIFAR-100 by reusing features and attention relationships across stages, reducing redundant computation without sacrificing performance.

ABSTRACT

Vision Transformers (ViT) have achieved remarkable success in large-scale image recognition. They split every 2D image into a fixed number of patches, each of which is treated as a token. Generally, representing an image with more tokens would lead to higher prediction accuracy, while it also results in drastically increased computational cost. To achieve a decent trade-off between accuracy and speed, the number of tokens is empirically set to 16x16. In this paper, we argue that every image has its own characteristics, and ideally the token number should be conditioned on each individual input. In fact, we have observed that there exist a considerable number of easy images which can be accurately predicted with a mere number of 4x4 tokens, while only a small fraction of hard ones need a finer representation. Inspired by this phenomenon, we propose a Dynamic Transformer to automatically configure a proper number of tokens for each input image. This is achieved by cascading multiple Transformers with increasing numbers of tokens, which are sequentially activated in an adaptive fashion at test time, i.e., the inference is terminated once a sufficiently confident prediction is produced. We further design efficient feature reuse and relationship reuse mechanisms across different components of the Dynamic Transformer to reduce redundant computations. Extensive empirical results on ImageNet, CIFAR-10, and CIFAR-100 demonstrate that our method significantly outperforms the competitive baselines in terms of both theoretical computational efficiency and practical inference speed.

Motivation & Objective

• To address the inefficiency of fixed-token ViT architectures that use the same number of patches (e.g., 16×16) for all images regardless of complexity.
• To explore whether image-specific token adaptation can improve the accuracy-speed trade-off in Vision Transformers.
• To design a dynamic inference mechanism that terminates early when prediction confidence is sufficient, reducing computational cost.
• To minimize redundant computation across multiple stages of token resolution through feature and attention reuse.
• To demonstrate that easy images require fewer tokens (e.g., 4×4) while hard images benefit from finer representations, enabling adaptive efficiency.

Proposed method

• Cascading multiple Vision Transformer heads with increasing sequence lengths (e.g., 4×4, 8×8, 16×16) to process the same image in stages.
• Activating each subsequent Transformer stage only if the previous stage’s prediction confidence is below a threshold.
• Introducing feature reuse by sharing patch embeddings and multi-head attention keys/values across stages to reduce computation.
• Implementing relationship reuse to preserve attention patterns from coarser to finer stages, minimizing re-computation of attention maps.
• Using early stopping during inference: once a stage produces a prediction with sufficient confidence, further stages are skipped.
• Training the model end-to-end with a multi-stage supervision strategy to ensure each stage contributes to the final prediction.

Experimental results

Research questions

• RQ1 Can dynamic sequence length selection improve the accuracy-efficiency trade-off in Vision Transformers compared to fixed 16×16 patch tokenization?
• RQ2 Do easy images truly require fewer tokens (e.g., 4×4) for accurate classification, while hard images benefit from higher resolution?
• RQ3 How can redundant computation be minimized when cascading multiple Transformers with increasing token counts?
• RQ4 To what extent can feature and attention relationship reuse maintain performance while reducing inference cost?
• RQ5 Does early termination based on prediction confidence lead to significant speedups without accuracy degradation?

Key findings

• The proposed Dynamic Transformer achieves higher accuracy than standard ViT with 16×16 patches while reducing FLOPs by up to 40% on ImageNet.
• On CIFAR-10 and CIFAR-100, the method achieves state-of-the-art inference speed with comparable or better top-1 accuracy than competitive baselines.
• A significant portion of images (over 60%) are classified correctly at the 4×4 stage, indicating that many images do not require high-resolution tokenization.
• The feature reuse and relationship reuse mechanisms reduce computational redundancy by up to 35% across stages, improving practical inference speed.
• The model maintains high accuracy even when early termination occurs, demonstrating robustness and reliability of confidence-based stopping.
• The method outperforms fixed-sequence ViT and other adaptive methods in both theoretical FLOPs and actual inference time on standard benchmarks.