[Paper Review] Global Attention Mechanism: Retain Information to Enhance Channel-Spatial Interactions

Yichao Liu, Zongru Shao+1 more|arXiv (Cornell University)|Dec 10, 2021

Advanced Neural Network ApplicationsComputer Science412 citations

TL;DR

The paper introduces Global Attention Mechanism (GAM), which preserves information across channel and spatial dimensions using 3D-permutation with an MLP and a convolutional spatial submodule, showing consistent gains over prior attention modules on CIFAR-100 and ImageNet-1K across ResNet and MobileNet backbones.

ABSTRACT

A variety of attention mechanisms have been studied to improve the performance of various computer vision tasks. However, the prior methods overlooked the significance of retaining the information on both channel and spatial aspects to enhance the cross-dimension interactions. Therefore, we propose a global attention mechanism that boosts the performance of deep neural networks by reducing information reduction and magnifying the global interactive representations. We introduce 3D-permutation with multilayer-perceptron for channel attention alongside a convolutional spatial attention submodule. The evaluation of the proposed mechanism for the image classification task on CIFAR-100 and ImageNet-1K indicates that our method stably outperforms several recent attention mechanisms with both ResNet and lightweight MobileNet.

Motivation & Objective

• Motivate the need to retain information across channel and spatial dimensions to enhance cross-dimension interactions in attention.
• Propose GAM to magnify global channel-spatial dependencies while reducing information loss.
• Evaluate GAM on standard benchmarks (CIFAR-100 and ImageNet-1K) across multiple architectures (ResNet and MobileNet) against existing attention modules.

Proposed method

• Channel attention submodule uses 3D permutation and a two-layer MLP to capture cross-dimension information.
• Spatial attention submodule uses two convolutional layers with no pooling to preserve information, with optional group convolution and channel shuffle to reduce parameters.
• GAM applies channel attention first, followed by spatial attention, with element-wise multiplication to form final feature maps.
• Comparisons are made against SE, BAM, CBAM, TAM, and ABN under the same training conditions.
• Ablation studies examine the contributions of channel vs. spatial attention and the effect of max-pooling in CBAM/GAM.

Experimental results

Research questions

• RQ1 Does GAM provide consistent performance gains over existing attention modules across datasets and architectures?
• RQ2 How do channel and spatial attention components contribute to GAM’s performance?
• RQ3 What is the impact of design choices (e.g., pooling, group convolution) on GAM’s efficiency and accuracy?
• RQ4 Can GAM scale effectively to large datasets and different model depths (ResNet18/50, MobileNetV2)?

Key findings

• GAM outperforms SE, BAM, and CBAM on CIFAR-100 with ResNet-50, including variants with group convolution.
• On ImageNet-1K, GAM consistently improves Top-1 and Top-5 errors across ResNet-18, ResNet-50, and MobileNetV2 compared to baseline and other attention modules.
• GAM achieves better accuracy with fewer parameters than some competitors (e.g., ResNet-18 with GAM vs ABN).
• Ablations show that both spatial and channel attention contribute to performance gains, and their combination yields the best results.
• Removing max-pooling in GAM’s design (or CBAM) can still yield strong performance, with GAM often maintaining advantages.