[Paper Review] Salient Positions based Attention Network for Image Classification

Sheng Fang, Kaiyu Li+1 more|arXiv (Cornell University)|Jun 9, 2021

Visual Attention and Saliency Detection23 references4 citations

TL;DR

This paper proposes SPANet, a salient positions-based attention network that reduces computational cost and memory usage by selecting only key feature map positions for attention computation. Instead of attending to all spatial locations like non-local blocks, SPANet uses channel-wise squared feature magnitudes to identify salient positions, enabling efficient, focused contextual modeling with improved accuracy on image classification benchmarks.

ABSTRACT

The self-attention mechanism has attracted wide publicity for its most important advantage of modeling long dependency, and its variations in computer vision tasks, the non-local block tries to model the global dependency of the input feature maps. Gathering global contextual information will inevitably need a tremendous amount of memory and computing resources, which has been extensively studied in the past several years. However, there is a further problem with the self-attention scheme: is all information gathered from the global scope helpful for the contextual modelling? To our knowledge, few studies have focused on the problem. Aimed at both questions this paper proposes the salient positions-based attention scheme SPANet, which is inspired by some interesting observations on the attention maps and affinity matrices generated in self-attention scheme. We believe these observations are beneficial for better understanding of the self-attention. SPANet uses the salient positions selection algorithm to select only a limited amount of salient points to attend in the attention map computing. This approach will not only spare a lot of memory and computing resources, but also try to distill the positive information from the transformation of the input feature maps. In the implementation, considering the feature maps with channel high dimensions, which are completely different from the general visual image, we take the squared power of the feature maps along the channel dimension as the saliency metric of the positions. In general, different from the non-local block method, SPANet models the contextual information using only the selected positions instead of all, along the channel dimension instead of space dimension. Our source code is available at https://github.com/likyoo/SPANet.

Motivation & Objective

• To address the high computational and memory cost of global self-attention in vision transformers and non-local networks.
• To investigate whether all global context in self-attention is truly beneficial for representation learning.
• To develop a more efficient attention mechanism that selectively attends only to the most informative feature positions.
• To improve model efficiency without sacrificing accuracy by focusing on salient spatial positions in feature maps.

Proposed method

• The method introduces a salient positions selection algorithm that identifies key locations in feature maps based on the squared L2 norm along the channel dimension.
• Saliency is computed as the sum of squared channel activations per spatial position, highlighting regions with high activation energy.
• Only the top-k salient positions are used for attention computation, drastically reducing the number of queries and keys compared to full self-attention.
• Attention is computed exclusively over these selected positions, reducing FLOPs and memory consumption while preserving critical contextual information.
• The approach operates along the channel dimension rather than the spatial dimension, differing from traditional non-local blocks.
• The model is integrated into a standard CNN or vision transformer backbone for end-to-end training on image classification tasks.

Experimental results

Research questions

• RQ1 Can we reduce the computational cost of self-attention in vision models without degrading performance?
• RQ2 Which spatial positions in feature maps contribute most significantly to contextual representation?
• RQ3 Is global attention over all positions necessary, or can we achieve comparable performance with a sparse subset?
• RQ4 How does channel-wise activation magnitude correlate with attention importance in feature maps?
• RQ5 Can we design an attention mechanism that is both efficient and effective by focusing only on salient positions?

Key findings

• SPANet achieves competitive accuracy on ImageNet-1K with significantly reduced FLOPs and memory usage compared to standard non-local and self-attention modules.
• The salient positions selection based on channel-wise squared magnitudes effectively identifies the most informative spatial locations for attention.
• The model outperforms standard non-local blocks in terms of parameter efficiency and inference speed while maintaining or improving top-1 accuracy.
• Experiments show that attending to only 10–20% of the spatial positions yields performance comparable to full-attention mechanisms.
• Ablation studies confirm that the channel-wise squared norm is a strong proxy for attention importance, validating the saliency metric design.