[Paper Review] Locally Enhanced Self-Attention: Combining Self-Attention and Convolution as Local and Context Terms

Chenglin Yang, Siyuan Qiao+2 more|arXiv (Cornell University)|Jul 12, 2021

Advanced Neural Network Applications52 references4 citations

TL;DR

This paper proposes Locally Enhanced Self-Attention (LESA), a novel attention mechanism that decomposes self-attention into local (unary) and context (binary) terms using shared query/key/value projections. It enhances the local term via grouped convolution and introduces a dynamic fusion module to adaptively combine both terms, achieving state-of-the-art performance on ImageNet, COCO detection, and instance segmentation with fewer parameters than baselines.

ABSTRACT

Self-Attention has become prevalent in computer vision models. Inspired by fully connected Conditional Random Fields (CRFs), we decompose self-attention into local and context terms. They correspond to the unary and binary terms in CRF and are implemented by attention mechanisms with projection matrices. We observe that the unary terms only make small contributions to the outputs, and meanwhile standard CNNs that rely solely on the unary terms achieve great performances on a variety of tasks. Therefore, we propose Locally Enhanced Self-Attention (LESA), which enhances the unary term by incorporating it with convolutions, and utilizes a fusion module to dynamically couple the unary and binary operations. In our experiments, we replace the self-attention modules with LESA. The results on ImageNet and COCO show the superiority of LESA over convolution and self-attention baselines for the tasks of image recognition, object detection, and instance segmentation. The code is made publicly available.

Motivation & Objective

• To address the limited contribution of local terms in standard self-attention, which are critical for precise feature representation but underutilized due to softmax normalization.
• To improve the representational capacity of the unary (local) term in self-attention by integrating it with grouped convolutions, overcoming the weakness of identity skip connections.
• To dynamically couple local and context terms using input-dependent attention weights, enabling adaptive fusion that improves performance across vision tasks.
• To demonstrate that combining convolution and self-attention through a parallel, learnable fusion mechanism yields better accuracy and efficiency than existing methods.

Proposed method

• Decompose self-attention into unary (local) and binary (context) terms using shared query, key, and value projections, inspired by fully connected CRFs.
• Enhance the unary term by replacing identity skip connections with grouped convolution, enabling richer local feature learning.
• Implement multi-head attention via grouped convolution followed by a projection layer to maintain multi-head capacity.
• Propose a dynamic fusion module that computes learnable, spatially and channel-wise adaptive weights ωˡ(x) to modulate the binary term before element-wise addition to the unary term.
• Use the same input features for both unary and binary terms, ensuring parameter efficiency while enabling end-to-end training.
• Replace spatial convolution layers in ResNet and WRN backbones with LESA in the last two stages, and evaluate on ImageNet, COCO detection, and instance segmentation.

Experimental results

Research questions

• RQ1 Can the local term in self-attention be significantly enhanced to improve feature representation without compromising global context modeling?
• RQ2 Does integrating convolutional operations into the unary term of self-attention lead to better performance than standard self-attention or convolution alone?
• RQ3 Can a dynamic, input-dependent fusion mechanism outperform static or fixed-weight fusion of local and context terms in vision models?
• RQ4 Is the proposed Locally Enhanced Self-Attention (LESA) more effective and parameter-efficient than existing hybrid convolution-self-attention architectures?

Key findings

• LESA achieves a top-1 accuracy of 79.55% on ImageNet using ResNet50, outperforming self-attention (78.69%) and Attention Augmentation (78.74%) with fewer parameters.
• On COCO object detection and instance segmentation, LESA with WRN50 backbone achieves higher mAP across all object scales (small, medium, large) than convolution, self-attention, and deformable convolution baselines.
• The ablation study confirms that the dynamic fusion module is essential, as LESA (79.55% top-1) outperforms Static LESA (79.12%) by a significant margin.
• LESA reduces the reliance on large similarity matrices by enhancing local features, which may help mitigate the memory inefficiency common in self-attention models.
• The model shows improved mask quality and object detail detection in instance segmentation, indicating better semantic consistency from the combined local-context learning.