[Paper Review] CvT: Introducing Convolutions to Vision Transformers

Haiping Wu, Bin Xiao+5 more|arXiv (Cornell University)|Mar 29, 2021

Advanced Neural Network Applications30 references162 citations

TL;DR

CvT integrates convolutions into Vision Transformers, creating a hierarchical, convolution-enhanced transformer that achieves state-of-the-art ImageNet results with fewer parameters and FLOPs, and even without positional embeddings.

ABSTRACT

We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (\ie shift, scale, and distortion invariance) while maintaining the merits of Transformers (\ie dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (\eg ImageNet-22k) and fine-tuned to downstream tasks. Pre-trained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7\% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks. Code will be released at \url{https://github.com/leoxiaobin/CvT}.

Motivation & Objective

• Motivate combining CNNs and Transformers to leverage local and global context for image recognition.
• Propose CvT with two core changes: convolutional token embedding and convolutional projection within a hierarchical transformer framework.
• Demonstrate improved accuracy and efficiency over ViT/DeiT and competitive CNNs on ImageNet-1k and ImageNet-22k.
• Show that positional encoding can be removed without performance loss, enabling variable input resolutions.

Proposed method

• Introduce a three-stage hierarchical CvT backbone that uses convolutional token embedding to progressively reduce token length while increasing feature dimension.
• Replace the attention input projection with a convolutional projection based on depthwise separable convolutions to model local context and enable token subsampling.
• Perform layer normalization after convolutional token embedding and apply a standard MLP head at the final stage for classification.
• Demonstrate that dropping positional embeddings does not degrade performance due to built-in local context from convolutions.

Figure 1 : Top-1 Accuracy on ImageNet validation compared to other methods with respect to model parameters. (a) Comparison to CNN-based model BiT [ 18 ] and Transformer-based model ViT [ 11 ] , when pretrained on ImageNet-22k. Larger marker size indicates larger architectures. (b) Comparison to con

Experimental results

Research questions

• RQ1 Does introducing convolutions into Vision Transformers improve accuracy and efficiency on large-scale image classification?
• RQ2 Can a hierarchical, multi-stage CvT architecture with convolutional token embedding and convolutional projections outperform ViT/DeiT and CNNs?
• RQ3 Is it possible to remove explicit positional embeddings without sacrificing performance when local context is captured by convolutions?
• RQ4 How do CvT models scale with dataset size (e.g., ImageNet-1k vs ImageNet-22k) and transfer to downstream tasks?

Key findings

• CvT-21 on ImageNet-1k achieves 82.5% top-1 with 7.1G FLOPs and 32M params, outperforming DeiT-B with fewer FLOPs/parameters.
• CvT-13 achieves 81.6% top-1 with 4.5G FLOPs and 20M params, surpassing several CNN/Transformer baselines in efficiency.
• CvT-W24 pretrained on ImageNet-22k reaches 87.7% top-1 on ImageNet-1k and maintains strong transfer results on CIFAR, PETS, and Flowers.
• Removing positional embeddings from CvT does not hurt performance, highlighting that convolutional components provide sufficient spatial bias.
• NAS-inspired variants (CvT-13-NAS) can achieve comparable accuracy with fewer parameters, indicating room for architecture search gains.
• CvT shows strong performance across resolutions (e.g., 384^2) with competitive or superior accuracy to other transformers at similar or lower FLOPs.

Figure 2 : The pipeline of the proposed CvT architecture. (a) Overall architecture, showing the hierarchical multi-stage structure facilitated by the Convolutional Token Embedding layer. (b) Details of the Convolutional Transformer Block, which contains the convolution projection as the first layer.