[Paper Review] Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios

Jiashi Li, Xin Xia+7 more|arXiv (Cornell University)|Jul 12, 2022

Advanced Neural Network ApplicationsComputer Science138 citations

TL;DR

Next-ViT introduces Next Convolution Block (NCB) and Next Transformer Block (NTB) with a Next Hybrid Strategy (NHS) to deliver CNN-competitive latency while matching ViT accuracy, outperforming existing models on industrial deployment platforms like TensorRT and CoreML.

ABSTRACT

Due to the complex attention mechanisms and model design, most existing vision Transformers (ViTs) can not perform as efficiently as convolutional neural networks (CNNs) in realistic industrial deployment scenarios, e.g. TensorRT and CoreML. This poses a distinct challenge: Can a visual neural network be designed to infer as fast as CNNs and perform as powerful as ViTs? Recent works have tried to design CNN-Transformer hybrid architectures to address this issue, yet the overall performance of these works is far away from satisfactory. To end these, we propose a next generation vision Transformer for efficient deployment in realistic industrial scenarios, namely Next-ViT, which dominates both CNNs and ViTs from the perspective of latency/accuracy trade-off. In this work, the Next Convolution Block (NCB) and Next Transformer Block (NTB) are respectively developed to capture local and global information with deployment-friendly mechanisms. Then, Next Hybrid Strategy (NHS) is designed to stack NCB and NTB in an efficient hybrid paradigm, which boosts performance in various downstream tasks. Extensive experiments show that Next-ViT significantly outperforms existing CNNs, ViTs and CNN-Transformer hybrid architectures with respect to the latency/accuracy trade-off across various vision tasks. On TensorRT, Next-ViT surpasses ResNet by 5.5 mAP (from 40.4 to 45.9) on COCO detection and 7.7% mIoU (from 38.8% to 46.5%) on ADE20K segmentation under similar latency. Meanwhile, it achieves comparable performance with CSWin, while the inference speed is accelerated by 3.6x. On CoreML, Next-ViT surpasses EfficientFormer by 4.6 mAP (from 42.6 to 47.2) on COCO detection and 3.5% mIoU (from 45.1% to 48.6%) on ADE20K segmentation under similar latency. Our code and models are made public at: https://github.com/bytedance/Next-ViT

Motivation & Objective

• Motivate the need for fast, deployment-friendly vision transformers in industrial scenarios (TensorRT/CoreML).
• Design blocks that combine local (NCB) and global (NTB) information efficiently.
• Propose a hybrid stacking strategy (NHS) to balance Transformer and Convolution blocks across stages.
• Demonstrate superior latency/accuracy trade-offs against CNNs, ViTs, and CNN-Transformer hybrids on downstream tasks.

Proposed method

• Develop Next-Convolution Block (NCB) with Multi-Head Convolutional Attention (MHCA) as a deployment-friendly token mixer.
• Develop Next Transformer Block (NTB) to capture multi-frequency signals via Efficient Multi-Head Self Attention (E-MHSA) and MHCA fusion.
• Introduce Next Hybrid Strategy (NHS) to stack NCB and NTB across stages in configurations like (NCB×N + NTB×1) per stage, with further repetition (×L) to improve performance under fixed latency.
• Use BatchNorm and ReLU instead of LayerNorm/GELU to accelerate inference on hardware like TensorRT/CoreML.
• Provide three Next-ViT variants (S/B/L) with specific stage configurations and channel settings (Table 3).
• Train and evaluate on ImageNet-1K classification and assess downstream tasks (COCO detection, ADE20K segmentation) under hardware-aware latency.

Experimental results

Research questions

• RQ1 Can a vision transformer be designed to infer as fast as CNNs while retaining ViT-level accuracy in realistic industrial deployments?
• RQ2 Do deployment-friendly blocks (NCB and NTB) and a hybrid strategy (NHS) improve latency/accuracy trade-offs across classification, detection, and segmentation tasks?
• RQ3 What is the impact of stacking patterns (NCB versus NTB in various stages) on throughput and task performance under TensorRT/CoreML constraints?

Key findings

• Next-ViT achieves the best latency/accuracy trade-off on ImageNet-1K classification among compared models.
• On TensorRT, Next-ViT surpasses ResNet by 5.5 mAP (from 40.4 to 45.9) on COCO detection and 7.7% mIoU (from 38.8% to 46.5%) on ADE20K segmentation under similar latency.
• Next-ViT achieves comparable performance with CSWin, while the inference speed is accelerated by 3.6×.
• On CoreML, Next-ViT surpasses EfficientFormer by 4.6 mAP (from 42.6 to 47.2) on COCO detection and 3.5% mIoU (from 45.1% to 48.6%) on ADE20K segmentation under similar latency.
• Results are demonstrated for Next-ViT-S/B/L variants with hardware-aware latency measurements (TensorRT/CoreML).