[Paper Review] CMT: Convolutional Neural Networks Meet Vision Transformers

Jianyuan Guo, Kai Han+5 more|arXiv (Cornell University)|Jul 13, 2021

Advanced Neural Network Applications65 references57 citations

TL;DR

CMT combines CNN stems with transformer-style blocks to create a hierarchical, multi-scale backbone that outperforms comparable CNN and ViT models on ImageNet and COCO with lower FLOPs.

ABSTRACT

Vision transformers have been successfully applied to image recognition tasks due to their ability to capture long-range dependencies within an image. However, there are still gaps in both performance and computational cost between transformers and existing convolutional neural networks (CNNs). In this paper, we aim to address this issue and develop a network that can outperform not only the canonical transformers, but also the high-performance convolutional models. We propose a new transformer based hybrid network by taking advantage of transformers to capture long-range dependencies, and of CNNs to model local features. Furthermore, we scale it to obtain a family of models, called CMTs, obtaining much better accuracy and efficiency than previous convolution and transformer based models. In particular, our CMT-S achieves 83.5% top-1 accuracy on ImageNet, while being 14x and 2x smaller on FLOPs than the existing DeiT and EfficientNet, respectively. The proposed CMT-S also generalizes well on CIFAR10 (99.2%), CIFAR100 (91.7%), Flowers (98.7%), and other challenging vision datasets such as COCO (44.3% mAP), with considerably less computational cost.

Motivation & Objective

• Motivate and address the performance-cost gap between CNNs and vision transformers.
• Propose a hybrid architecture that leverages CNNs for local feature extraction and transformers for long-range dependencies.
• Design a scalable family of CMT models with stage-wise structure and efficiency-focused blocks.

Proposed method

• Introduce a convolution stem to extract fine-grained features before CMT blocks.
• Develop a CMT block with Local Perception Unit (LPU), Lightweight Multi-head Self-Attention (LMHSA), and Inverted Residual Feed-Forward Network (IRFFN).
• Use depth-wise convolutions and a lightweight attention with spatial reduction to reduce compute.
• Apply a stage-wise architecture to produce multi-scale feature maps suitable for dense tasks.
• Adopt a compound scaling strategy to generate model variants (CMT-Ti, XS, S, B, etc.).
• Replace ViT’s class token with global average pooling for classification.

Experimental results

Research questions

• RQ1 Can a CNN-augmented transformer backbone surpass pure CNN and pure transformer models in accuracy and efficiency on ImageNet and COCO?
• RQ2 How do local and global information interplay in a hybrid CMT block to improve representation learning?
• RQ3 What is the impact of stage-wise design and the proposed scaling strategy on performance and FLOPs?
• RQ4 Does the CMT backbone transfer well to object detection and instance segmentation tasks?

Key findings

• CMT-S achieves 83.5% top-1 accuracy on ImageNet with 4.0B FLOPs, outperforming DeiT-S and CPVT while using less computation than EfficientNet-B4.
• CMT variants demonstrate strong performance across datasets (CIFAR-10/100, Flowers, COCO) with favorable accuracy/FLOPs trade-offs.
• Stage-wise architecture and LMHSA reduce computation while maintaining or improving accuracy compared to single-scale ViT backbones.
• A convolutional stem, LPU, IRFFN, and short-cut connections in CMT blocks significantly boost accuracy (e.g., +0.9% to +1.1% in ablations).
• On COCO, CMT-S achieves 44.3% mAP with RetinaNet, surpassing several CNN/ViT-based backbones at similar FLOPs.
• Transfer learning experiments show competitive performance across CIFAR and fine-grained datasets when pretrained on ImageNet.