[Paper Review] Attention to Scale: Scale-aware Semantic Image Segmentation

Liang-Chieh Chen, Yi Yang+3 more|arXiv (Cornell University)|Nov 10, 2015

Advanced Neural Network Applications5 references134 citations

TL;DR

This paper proposes a scale-aware attention mechanism for semantic image segmentation that dynamically weights multi-scale features from a shared fully convolutional network, improving performance over average- and max-pooling baselines. The method jointly trains with extra supervision at each scale, achieving state-of-the-art results on PASCAL-Person-Part, PASCAL VOC 2012, and MS-COCO, while enabling diagnostic visualization of scale-specific feature importance.

ABSTRACT

Incorporating multi-scale features in fully convolutional neural networks (FCNs) has been a key element to achieving state-of-the-art performance on semantic image segmentation. One common way to extract multi-scale features is to feed multiple resized input images to a shared deep network and then merge the resulting features for pixelwise classification. In this work, we propose an attention mechanism that learns to softly weight the multi-scale features at each pixel location. We adapt a state-of-the-art semantic image segmentation model, which we jointly train with multi-scale input images and the attention model. The proposed attention model not only outperforms average- and max-pooling, but allows us to diagnostically visualize the importance of features at different positions and scales. Moreover, we show that adding extra supervision to the output at each scale is essential to achieving excellent performance when merging multi-scale features. We demonstrate the effectiveness of our model with extensive experiments on three challenging datasets, including PASCAL-Person-Part, PASCAL VOC 2012 and a subset of MS-COCO 2014.

Motivation & Objective

• Address the challenge of handling objects at varying scales in semantic image segmentation using deep learning.
• Overcome limitations of fixed merging strategies like average- or max-pooling that treat all scales uniformly.
• Improve segmentation accuracy by learning adaptive, dynamic weighting of multi-scale features based on object scale.
• Enable diagnostic visualization of which scales contribute most to predictions at each spatial location.
• Demonstrate the necessity of extra supervision at each scale for optimal performance in multi-scale fusion.

Proposed method

• Adapts the DeepLab-LargeFOV model as a backbone for multi-scale input processing via a share-net architecture.
• Applies a soft attention mechanism that learns a spatially varying weight map for each scale, dynamically modulating feature responses.
• Fuses multi-scale features by computing a weighted sum of score maps using attention weights, replacing fixed pooling operations.
• Introduces extra supervision to the final output of the network at each scale to improve feature learning and fusion.
• Jointly trains the attention module and the segmentation network end-to-end using multi-scale inputs and scale-specific supervision.
• Employs a learnable attention module that attends to scale-specific features based on local context and object size.

Experimental results

Research questions

• RQ1 Can a learnable attention mechanism improve multi-scale feature fusion in semantic segmentation beyond fixed pooling strategies?
• RQ2 How does the attention mechanism adaptively assign importance to features from different scales based on object size?
• RQ3 What is the impact of adding extra supervision at each scale on segmentation performance?
• RQ4 Can the attention mechanism provide meaningful diagnostic insights into which scales contribute most to predictions?
• RQ5 Does the proposed method generalize across diverse datasets with varying object scale distributions?

Key findings

• The proposed attention mechanism achieves 35.41% mean IoU on PASCAL VOC 2012 with scales {1, 0.5}, outperforming max-pooling (34.70%) and average-pooling (35.14%).
• On the subset of MS-COCO 2014, the method achieves 35.78% mean IoU with scales {1, 0.75, 0.5}, showing a 4.6% improvement over the DeepLab-LargeFOV baseline (31.22%).
• For the person class on MS-COCO, the attention model achieves 72.72% IoU, a 1.02% improvement over average-pooling and 2.06% over max-pooling.
• The attention mechanism enables diagnostic visualization, revealing that small objects receive higher attention weights from finer-scale features, while large objects are better captured at coarser scales.
• Extra supervision at each scale is essential for performance, as ablation shows significant drops when omitted.
• The method generalizes across datasets, achieving consistent improvements on PASCAL-Person-Part, PASCAL VOC 2012, and MS-COCO, demonstrating robustness to scale variance and class imbalance.