[Paper Review] ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases

Stéphane d’Ascoli, Hugo Touvron+4 more|arXiv (Cornell University)|Mar 19, 2021

Advanced Neural Network Applications47 references57 citations

TL;DR

Introduces gated positional self-attention (GPSA) to softly inject convolutional inductive bias into Vision Transformers, improving sample efficiency and ImageNet performance over DeiT without external data pretraining.

ABSTRACT

Convolutional architectures have proven extremely successful for vision tasks. Their hard inductive biases enable sample-efficient learning, but come at the cost of a potentially lower performance ceiling. Vision Transformers (ViTs) rely on more flexible self-attention layers, and have recently outperformed CNNs for image classification. However, they require costly pre-training on large external datasets or distillation from pre-trained convolutional networks. In this paper, we ask the following question: is it possible to combine the strengths of these two architectures while avoiding their respective limitations? To this end, we introduce gated positional self-attention (GPSA), a form of positional self-attention which can be equipped with a ``soft" convolutional inductive bias. We initialise the GPSA layers to mimic the locality of convolutional layers, then give each attention head the freedom to escape locality by adjusting a gating parameter regulating the attention paid to position versus content information. The resulting convolutional-like ViT architecture, ConViT, outperforms the DeiT on ImageNet, while offering a much improved sample efficiency. We further investigate the role of locality in learning by first quantifying how it is encouraged in vanilla self-attention layers, then analysing how it is escaped in GPSA layers. We conclude by presenting various ablations to better understand the success of the ConViT. Our code and models are released publicly at https://github.com/facebookresearch/convit.

Motivation & Objective

• Motivate bridging CNNs and ViTs by introducing a soft convolutional inductive bias.
• Develop GPSA layers that can initialize as convolutional and gradually rely more on content.
• Show ConViT improves over DeiT in accuracy and sample efficiency without extra data.
• Analyze how locality is learned and escaped in GPSA versus vanilla self-attention.
• Provide ablations to understand the role of initialization, gating, and GPSA placement.

Proposed method

• Define gated positional self-attention (GPSA) that combines content and positional terms with a learnable gate λ_h.
• Initialize GPSA to mimic a convolutional kernel (convolutional initialization) and keep relative positional encodings fixed.
• Introduce a gating mechanism to blend position-based and content-based attention after the softmax (Eq. 7).
• Replace a subset of ViT self-attention layers with GPSA layers in a DeiT-based architecture to create ConViT.
• Analyze locality dynamics via a nonlocality metric and inspect gating parameters across layers (and heads).
• Provide open-source code and pretrained models for reproducibility.

Experimental results

Research questions

• RQ1 Can a soft, learnable convolutional inductive bias be integrated into Vision Transformers without locking the model into a fixed CNN-like regime?
• RQ2 How does localization emerge in vanilla self-attention, and how do GPSA layers escape locality during training?
• RQ3 Do GPSA-based ConViTs offer improved sample efficiency and competitive accuracy compared to DeiT, especially in data-limited regimes?

Key findings

• ConViT outperforms DeiT of the same size and compute, with higher Top-1 and Top-5 on ImageNet in several configurations.
• ConViT-S+ achieves 82.2% top-1 (vs 81.4% for DeiT-S), with improved throughput relative to some baselines.
• GPSA layers provide a soft, controllable convolutional bias that enhances early training dynamics and sample efficiency.
• Gating parameters reveal heads that pay more to positional information in early layers and shift toward content in later layers.
• Ablations show the convolutional initialization and gating jointly contribute to gains, especially under low-data conditions.