[Paper Review] MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features

Shakti N. Wadekar, Abhishek Chaurasia|arXiv (Cornell University)|Sep 30, 2022

Advanced Neural Network Applications86 citations

TL;DR

MobileViTv3 reintroduces and simplifies a fusion block in MobileViT to fuse local, global, and input features, enabling scalable, efficient, mobile-friendly vision transformers with improved accuracy across ImageNet-1K, ADE20K, COCO, and Pascal VOC.

ABSTRACT

MobileViT (MobileViTv1) combines convolutional neural networks (CNNs) and vision transformers (ViTs) to create light-weight models for mobile vision tasks. Though the main MobileViTv1-block helps to achieve competitive state-of-the-art results, the fusion block inside MobileViTv1-block, creates scaling challenges and has a complex learning task. We propose changes to the fusion block that are simple and effective to create MobileViTv3-block, which addresses the scaling and simplifies the learning task. Our proposed MobileViTv3-block used to create MobileViTv3-XXS, XS and S models outperform MobileViTv1 on ImageNet-1k, ADE20K, COCO and PascalVOC2012 datasets. On ImageNet-1K, MobileViTv3-XXS and MobileViTv3-XS surpasses MobileViTv1-XXS and MobileViTv1-XS by 2% and 1.9% respectively. Recently published MobileViTv2 architecture removes fusion block and uses linear complexity transformers to perform better than MobileViTv1. We add our proposed fusion block to MobileViTv2 to create MobileViTv3-0.5, 0.75 and 1.0 models. These new models give better accuracy numbers on ImageNet-1k, ADE20K, COCO and PascalVOC2012 datasets as compared to MobileViTv2. MobileViTv3-0.5 and MobileViTv3-0.75 outperforms MobileViTv2-0.5 and MobileViTv2-0.75 by 2.1% and 1.0% respectively on ImageNet-1K dataset. For segmentation task, MobileViTv3-1.0 achieves 2.07% and 1.1% better mIOU compared to MobileViTv2-1.0 on ADE20K dataset and PascalVOC2012 dataset respectively. Our code and the trained models are available at: https://github.com/micronDLA/MobileViTv3

Motivation & Objective

• Motivate efficient vision models for mobile/edge devices that balance accuracy and resource usage.
• Improve the fusion mechanism in MobileViT blocks to enable scalable architectures with low parameter and compute counts.
• Demonstrate that adding a lightweight fusion block to MobileViTv2 yields superior performance across classification, segmentation, and detection tasks.

Proposed method

• Replace the 3x3 fusion conv with a 1x1 conv in the MobileViT fusion block to decouple fusion from spatial context.
• Fuse local and global representations by concatenating them in the fusion stage instead of fusing input and global features.
• Add the input features to the fusion block output as a residual connection before producing the final MobileViT block output.
• Replace the 3x3 conv in the local representation block with a depthwise 3x3 conv to reduce parameters.
• Scale MobileViTv3 by widening (increasing channels) to create MobileViTv3-S, XS, XXS models while maintaining similar FLOPs.

Experimental results

Research questions

• RQ1 Can the MobileViT fusion block be redesigned to reduce parameter growth while preserving or improving accuracy on mobile hardware?
• RQ2 Does fusing local and global features (instead of input and global features) plus a residual input addition improve learning and performance?
• RQ3 What are the trade-offs between parameter count, FLOPs, accuracy, and latency when widening MobileViTv3 blocks for different deployment budgets?

Key findings

• MobileViTv3-XXS and MobileViTv3-XS surpass MobileViTv1-XXS and MobileViTv1-XS on ImageNet-1K by 2.0% and 1.9% with similar parameters/FLOPs.
• MobileViTv3-0.5 and MobileViTv3-0.75 outperform MobileViTv2-0.5 and MobileViTv2-0.75 by 2.1% and 1.0% on ImageNet-1K, respectively.
• On segmentation, MobileViTv3-1.0 achieves +2.07% mIOU on ADE20K and +1.1% on Pascal VOC 2012 over MobileViTv2-1.0.
• For detection (COCO), MobileViTv3-XS and MobileViTv3-1.0 achieve +0.8% and +0.5% mAP over MobileViTv1-XS and MobileViTv2-1.0, respectively.
• Latency and throughput improvements are demonstrated by reducing blocks in layer4; e.g., MobileViTv3-XXS with two blocks achieves ~6.24 ms latency vs ~7.24 ms for MobileViTv1-XXS at similar accuracy and parameters.