[Paper Review] Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks

Xiang Li, Xiaolin Hu+1 more|arXiv (Cornell University)|May 23, 2019

Advanced Neural Network Applications44 references178 citations

TL;DR

SGE introduces a lightweight spatial group-wise attention that enhances semantic sub-features inside each group by using a global-local similarity to generate per-location attention, boosting performance with negligible overhead. It improves classification and detection across backbones like ResNet and detectors on COCO.

ABSTRACT

The Convolutional Neural Networks (CNNs) generate the feature representation of complex objects by collecting hierarchical and different parts of semantic sub-features. These sub-features can usually be distributed in grouped form in the feature vector of each layer, representing various semantic entities. However, the activation of these sub-features is often spatially affected by similar patterns and noisy backgrounds, resulting in erroneous localization and identification. We propose a Spatial Group-wise Enhance (SGE) module that can adjust the importance of each sub-feature by generating an attention factor for each spatial location in each semantic group, so that every individual group can autonomously enhance its learnt expression and suppress possible noise. The attention factors are only guided by the similarities between the global and local feature descriptors inside each group, thus the design of SGE module is extremely lightweight with \emph{almost no extra parameters and calculations}. Despite being trained with only category supervisions, the SGE component is extremely effective in highlighting multiple active areas with various high-order semantics (such as the dog's eyes, nose, etc.). When integrated with popular CNN backbones, SGE can significantly boost the performance of image recognition tasks. Specifically, based on ResNet50 backbones, SGE achieves 1.2\% Top-1 accuracy improvement on the ImageNet benchmark and 1.0$\sim$2.0\% AP gain on the COCO benchmark across a wide range of detectors (Faster/Mask/Cascade RCNN and RetinaNet). Codes and pretrained models are available at https://github.com/implus/PytorchInsight.

Motivation & Objective

• Motivate improved learning of semantic sub-features within grouped CNN channels.
• Propose a lightweight module that enhances spatial distribution of group features without extensive parameters.
• Show that SGE improves localization of semantic regions and reduces noise in feature maps.
• Demonstrate improved performance on image classification and object detection benchmarks.

Proposed method

• Divide feature maps into G channel groups and treat each as a semantic group.
• Compute a global group feature g by averaging spatial locations within a group.
• Compute per-location coefficients c_i by dot product of g with local x_i, then normalize to obtain a_i via learnable gamma and beta.
• Scale x_i by a sigmoid gate sigma(a_i) to produce enhanced features hat{x}_i within each group.
• Integrate SGE after BatchNorm in bottlenecks with negligible extra parameters (gamma, beta per group).
• Provide visualization and ablation studies to validate normalization, group number, and initialization effects.

Experimental results

Research questions

• RQ1 Does the Spatial Group-wise Enhance module reliably boost semantic feature learning within groups?
• RQ2 Can SGE improve performance on image classification and object detection with minimal parameter overhead?
• RQ3 How do design choices (group count, normalization, initialization) affect SGE effectiveness?
• RQ4 How does SGE compare to existing attention modules in terms of performance and efficiency?

Key findings

• SGE yields 1.2% Top-1 accuracy improvement on ImageNet with ResNet50 and 1.0–2.0% AP gains on COCO detectors across Faster/Mask/Cascade RCNN and RetinaNet.
• SGE consistently improves small-object detection, outperforming SE in RetinaNet for small objects by about 1% AP.
• SGE provides competitive or superior results to state-of-the-art attention modules with fewer parameters and lower computational cost.
• Normalization is essential for stable learning and performance; removing normalization significantly degrades accuracy.
• Increasing then decreasing the number of groups G shows a sweet spot (commonly G = 32 or 64) for best performance.