[论文解读] EfficientFormer: Vision Transformers at MobileNet Speed

Yanyu Li, Geng Yuan+6 more|arXiv (Cornell University)|Jun 2, 2022

Advanced Neural Network Applications被引用 251

一句话总结

EfficientFormer 展示了在移动设备上以 MobileNet 类速度运行、同时在 ImageNet 上获得高准确度的纯视觉转换器，通过一个以延迟驱动、维度一致的设计和瘦身实现。

ABSTRACT

Vision Transformers (ViT) have shown rapid progress in computer vision tasks, achieving promising results on various benchmarks. However, due to the massive number of parameters and model design, extit{e.g.}, attention mechanism, ViT-based models are generally times slower than lightweight convolutional networks. Therefore, the deployment of ViT for real-time applications is particularly challenging, especially on resource-constrained hardware such as mobile devices. Recent efforts try to reduce the computation complexity of ViT through network architecture search or hybrid design with MobileNet block, yet the inference speed is still unsatisfactory. This leads to an important question: can transformers run as fast as MobileNet while obtaining high performance? To answer this, we first revisit the network architecture and operators used in ViT-based models and identify inefficient designs. Then we introduce a dimension-consistent pure transformer (without MobileNet blocks) as a design paradigm. Finally, we perform latency-driven slimming to get a series of final models dubbed EfficientFormer. Extensive experiments show the superiority of EfficientFormer in performance and speed on mobile devices. Our fastest model, EfficientFormer-L1, achieves $79.2\%$ top-1 accuracy on ImageNet-1K with only $1.6$ ms inference latency on iPhone 12 (compiled with CoreML), which runs as fast as MobileNetV2$ imes 1.4$ ($1.6$ ms, $74.7\%$ top-1), and our largest model, EfficientFormer-L7, obtains $83.3\%$ accuracy with only $7.0$ ms latency. Our work proves that properly designed transformers can reach extremely low latency on mobile devices while maintaining high performance.

研究动机与目标

• Identify bottlenecks and inefficiencies in ViT designs on mobile hardware.
• Propose a dimension-consistent transformer design that preserves speed and performance on edge devices.
• Develop a latency-driven slimming strategy to derive a family of EfficientFormer models.
• Demonstrate EfficientFormer as a backbone for detection and segmentation tasks.
• Provide a reproducible evaluation on mobile (iPhone) and desktop hardware.

提出的方法

• Perform on-device latency analysis of ViT components on iPhone 12 with CoreML to identify bottlenecks.
• Introduce a dimension-consistent design with a 4D Conv-BN based block (MB4D) and a 3D MHSA-based block (MB3D).
• Define a patch embedding stem with fast downsampling and avoid large-kernel patch embedding as a speed bottleneck.
• Construct a supernet with MetaBlocks allowing 4D and 3D configurations and use a gradient-based, latency-aware slimming procedure guided by a latency lookup table to obtain EfficientFormer variants.
• Train the supernet and apply Gumbel-Softmax sampling to estimate block importances, followed by latency-driven pruning/width adjustment to meet target latency.
• Evaluate EfficientFormer on ImageNet-1K, COCO, and ADE20K to showcase classification, object detection, and segmentation performance.

实验结果

研究问题

• RQ1 Can vision transformers be run at mobile-speed latency comparable to MobileNet while maintaining high accuracy?
• RQ2 What design choices in ViT architectures most impact on-device latency, and how can we restructure transformers to maximize speed on edge hardware?
• RQ3 Does a latency-driven slimming pipeline using a dimension-consistent transformer design yield practical, high-performance mobile models?
• RQ4 How do EfficientFormer backbones perform in detection and segmentation benchmarks relative to CNN and hybrid ViT designs?

主要发现

• EfficientFormer-L1 achieves 79.2% top-1 accuracy on ImageNet-1K with 1.6 ms latency on iPhone 12 (CoreML), comparable to MobileNetV2×1.4 speed.
• EfficientFormer-L7 reaches 83.3% top-1 accuracy with 7.0 ms latency on iPhone 12.
• On Nvidia A100, EfficientFormer-L1 is 38% faster than EfficientNet-B0 with 2.1% higher top-1; L7 is 4.6× faster than EfficientNet-B5.
• EfficientFormer outperforms PoolFormer and other ViT-based models in latency and accuracy, demonstrating strong edge performance of pure transformer architectures.
• Backbone experiments on COCO (detection/instance segmentation) and ADE20K (segmentation) show EfficientFormer-L3/L7 outperform ResNet and PoolFormer baselines under similar computation budgets.
• The dimension-consistent MB4D/MB3D design and latency-driven slimming enable ultra-fast transformer inference on mobile hardware.