[Paper Review] RepViT: Revisiting Mobile CNN From ViT Perspective

Ao Wang, Hui Chen+3 more|arXiv (Cornell University)|Jul 18, 2023

Robotics and Automated SystemsEngineering27 citations

TL;DR

RepViT shows that a pure lightweight CNN, when modernized with ViT-inspired architectural choices, can outperform lightweight ViTs on mobile devices in accuracy and latency, achieving over 80% top-1 on ImageNet with 1ms latency on iPhone 12 for M1-size models.

ABSTRACT

Recently, lightweight Vision Transformers (ViTs) demonstrate superior performance and lower latency, compared with lightweight Convolutional Neural Networks (CNNs), on resource-constrained mobile devices. Researchers have discovered many structural connections between lightweight ViTs and lightweight CNNs. However, the notable architectural disparities in the block structure, macro, and micro designs between them have not been adequately examined. In this study, we revisit the efficient design of lightweight CNNs from ViT perspective and emphasize their promising prospect for mobile devices. Specifically, we incrementally enhance the mobile-friendliness of a standard lightweight CNN, \ie, MobileNetV3, by integrating the efficient architectural designs of lightweight ViTs. This ends up with a new family of pure lightweight CNNs, namely RepViT. Extensive experiments show that RepViT outperforms existing state-of-the-art lightweight ViTs and exhibits favorable latency in various vision tasks. Notably, on ImageNet, RepViT achieves over 80\% top-1 accuracy with 1.0 ms latency on an iPhone 12, which is the first time for a lightweight model, to the best of our knowledge. Besides, when RepViT meets SAM, our RepViT-SAM can achieve nearly 10$ imes$ faster inference than the advanced MobileSAM. Codes and models are available at \url{https://github.com/THU-MIG/RepViT}.

Motivation & Objective

• Assess the limitations of current lightweight CNNs versus lightweight ViTs on mobile devices.
• Explore ViT-inspired architectural choices to modernize MobileNetV3-L into a pure CNN backbone.
• Demonstrate that RepViT achieves superior latency-accuracy on ImageNet and transfers well to downstream tasks.

Proposed method

• Start from MobileNetV3-L and progressively incorporate ViT-inspired design principles.
• Introduce RepViT block by separating token mixer and channel mixer with structural re-parameterization.
• Apply macro architectural tweaks: stem with early convolutions, deeper downsampling, simplified classifier, and optimized stage ratios.
• Conduct micro-architectural refinements: kernel size normalization to 3x3 and cross-block SE placement.
• Train and evaluate all models on ImageNet-1K; measure on-device latency on iPhone 12 with Core ML Tools; validate on COCO and ADE20K.

Figure 1 : Comparison of latency and accuracy between RepViT (Ours) and other lightweight models. The top-1 accuracy is tested on ImageNet-1K and the latency is measured by iPhone 12 with iOS 16. RepViT achieves high performance with low latency across various model sizes.

Experimental results

Research questions

• RQ1 Can architectural choices from lightweight ViTs improve the performance and latency of pure CNNs for mobile devices?
• RQ2 What macro- and micro-design adjustments best bridge CNN and ViT efficiency on edge devices?
• RQ3 How does RepViT perform on ImageNet and transfer to downstream tasks compared with state-of-the-art lightweight ViTs and CNNs?

Key findings

• RepViT outperforms existing state-of-the-art lightweight ViTs and CNNs in latency and accuracy across model sizes.
• RepViT-M0.9 to RepViT-M2.3 achieve strong ImageNet results with substantially lower on-device latency on iPhone 12 (e.g., 1 ms for small variants and 2.3 ms for larger variants).
• RepViT-M1.0 exceeds 80% top-1 accuracy with 1 ms latency on iPhone 12; RepViT-M2.3 reaches 83.7% accuracy with 2.3 ms latency.
• Downstream tasks (COCO object detection/segmentation and ADE20K semantic segmentation) show RepViT backbones achieving competitive AP and mIoU with lower latency than many competitors.
• Structural re-parameterization and cross-block SE placement consistently improve accuracy-latency trade-offs.
• RepViT demonstrates that pure lightweight CNNs can outperform lightweight ViTs on mobile devices when ViT-inspired architectural principles are integrated.

Figure 2 : We modernize MobileNetV3-L from various granularities. We mainly consider the latency on mobile devices and the top-1 accuracy on ImageNet-1K. Finally, we obtain a new family of pure lightweight CNNs, namely RepViT, which can achieve lower latency and higher performance.