[论文解读] Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition

Yulin Wang, Rui Huang+3 more|arXiv (Cornell University)|May 31, 2021

Advanced Neural Network Applications被引用 70

一句话总结

本文提出 Dynamic Vision Transformer (DVT) ，通过级联变换器并使用提前退出，按输入自适应选择令牌数量，并结合特征与关系重用，在不牺牲准确性的前提下提高效率。

ABSTRACT

Vision Transformers (ViT) have achieved remarkable success in large-scale image recognition. They split every 2D image into a fixed number of patches, each of which is treated as a token. Generally, representing an image with more tokens would lead to higher prediction accuracy, while it also results in drastically increased computational cost. To achieve a decent trade-off between accuracy and speed, the number of tokens is empirically set to 16x16 or 14x14. In this paper, we argue that every image has its own characteristics, and ideally the token number should be conditioned on each individual input. In fact, we have observed that there exist a considerable number of "easy" images which can be accurately predicted with a mere number of 4x4 tokens, while only a small fraction of "hard" ones need a finer representation. Inspired by this phenomenon, we propose a Dynamic Transformer to automatically configure a proper number of tokens for each input image. This is achieved by cascading multiple Transformers with increasing numbers of tokens, which are sequentially activated in an adaptive fashion at test time, i.e., the inference is terminated once a sufficiently confident prediction is produced. We further design efficient feature reuse and relationship reuse mechanisms across different components of the Dynamic Transformer to reduce redundant computations. Extensive empirical results on ImageNet, CIFAR-10, and CIFAR-100 demonstrate that our method significantly outperforms the competitive baselines in terms of both theoretical computational efficiency and practical inference speed. Code and pre-trained models (based on PyTorch and MindSpore) are available at https://github.com/blackfeather-wang/Dynamic-Vision-Transformer and https://github.com/blackfeather-wang/Dynamic-Vision-Transformer-MindSpore.

研究动机与目标

• 通过展示固定令牌的 ViTs 在简单图像上浪费计算来推动高效的图像识别。
• 提出一个动态令牌配置框架，按输入情况自适应以平衡精度和效率。
• 开发用于重用特征和自注意力关系以减少冗余计算的机制。
• 在标准基准和不同骨干网络上证明效率和准确性的提升。

提出的方法

• 以级联多个 Transformer，令牌数量逐步增多（例如 7x7、10x10、14x14）。
• 早停：当预测置信度超过学习阈值时停止推理。
• 以联合目标训练，汇总所有出口的交叉熵损失。
• 特征重用：将上游最终特征的上下文嵌入注入到下游 MLP 中。
• 关系重用：将上游自注意力的 logits 转移并调整以细化下游注意力图（A_l + r_l(A_up))。
• 通过调整瓶颈前向传递以实现在线可变 FLOPs，同时保持准确性。

实验结果

研究问题

• RQ1 按输入分配令牌预算是否可以在不牺牲准确性的前提下提高 ViT 的效率？
• RQ2 我们如何在级联变换器之间重用特征和注意力，以避免冗余计算？
• RQ3 自适应令牌数量和早停对实际推理速度和 FLOPs 有何影响？
• RQ4 DVT 的变体在不同骨干（例如 T2T-ViT、DeiT）和数据集（ImageNet、CIFAR-10/100）上的表现如何？

主要发现

• DVT 在 ImageNet 上持续降低骨干网络的计算成本，同时在保持或提高准确性的情况下提升性能。
• 在 ImageNet 上，DVT 在与原生骨干相当的 FLOPs 下实现更高的准确性，或在相近准确度下大幅减少 FLOPs（例如少于 3.6x 的 FLOPs）。
• 两种重用机制（特征重用和关系重用）在额外少量计算的情况下显著提升退出准确度。
• DVT 在 GPU 上带来实际的加速（例如提高吞吐量和真实推理速度），并在相似 FLOPs 下超越若干最先进的高效方法。
• 消融分析表明两种重用机制都对性能有贡献，当两者共同使用时获得累积增益。