[Paper Review] On the Relationship between Self-Attention and Convolutional Layers

Jean-Baptiste Cordonnier, Andreas Loukas+1 more|arXiv (Cornell University)|Nov 8, 2019

Advanced Neural Network Applications89 citations

TL;DR

The paper proves that multi-head self-attention with relative positional encoding can express any convolutional layer, and empirically shows attention heads learn grid-like, CNN-like local patterns in early layers.

ABSTRACT

Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks. This raises the question: do learned attention layers operate similarly to convolutional layers? This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Our numerical experiments then show that self-attention layers attend to pixel-grid patterns similarly to CNN layers, corroborating our analysis. Our code is publicly available.

Motivation & Objective

• Motivate the question of whether self-attention layers process images similarly to convolutional layers.
• Provide theoretical conditions under which self-attention can replicate convolutional layers.
• Empirically verify that attention heads learn localized, grid-like patterns in early layers of attention-only networks.

Proposed method

• Provide a constructive proof that a multi-head self-attention layer with relative positional encoding can express any convolutional layer (Theorem 1).
• Show that with quadratic or learned relative positional encodings, attention heads attend to grid-like patterns around each query pixel.
• Derive sufficient conditions (Lemmas 1 and 2) for re-parametrizing convolution using self-attention.
• Experiment with six-layer self-attention models on CIFAR-10 to observe attention patterns and compare accuracy to ResNet-18.
• Analyze both quadratic and learned relative positional encodings, including content-based attention, to study locality versus global dependencies.

Experimental results

Research questions

• RQ1 Can multi-head self-attention with relative positional encoding replicate the functional behavior of convolutional layers?
• RQ2 Do attention heads in practice learn localized, convolution-like receptive fields in early layers when trained on image data?
• RQ3 How do different positional encoding schemes (quadratic vs learned relative encoding) influence the localization and content-based attention in vision transformers?
• RQ4 What is the empirical performance and computational cost trade-off when using attention-only architectures versus traditional CNNs on standard image classification tasks?

Key findings

• A multi-head self-attention layer with enough heads and relative positional encoding can express any convolutional layer (kernel size K×K, min(Dh, Dout) output channels).
• In practice, early attention layers learn to attend to grid-like, localized patterns around the query pixel, resembling convolutional receptive fields.
• Quadratic and learned relative encodings both yield localized attention patterns, with later layers attending to larger patterns and incorporating content-based attention in deeper layers.
• On CIFAR-10, the attention models reach competitive accuracy with ResNet-18 under similar parameter counts, though some configurations lag (e.g., learned embedding with content attention).
• Attention probabilities in early layers tend to avoid overlap and cover the input space with diverse localized patterns, while deeper layers exhibit more global, content-driven attention.