[Paper Review] Hierarchical Multi-Scale Attention for Semantic Segmentation

Andrew Tao, Karan Sapra+1 more|arXiv (Cornell University)|May 21, 2020

Advanced Neural Network ApplicationsComputer Science42 references346 citations

TL;DR

The paper introduces a hierarchical multi-scale attention mechanism to combine multi-scale semantic segmentation predictions, improving accuracy while being memory-efficient and flexible at inference; it achieves state-of-the-art results on Cityscapes and Mapillary Vistas and utilizes hard auto-labelling for Cityscapes.

ABSTRACT

Multi-scale inference is commonly used to improve the results of semantic segmentation. Multiple images scales are passed through a network and then the results are combined with averaging or max pooling. In this work, we present an attention-based approach to combining multi-scale predictions. We show that predictions at certain scales are better at resolving particular failures modes, and that the network learns to favor those scales for such cases in order to generate better predictions. Our attention mechanism is hierarchical, which enables it to be roughly 4x more memory efficient to train than other recent approaches. In addition to enabling faster training, this allows us to train with larger crop sizes which leads to greater model accuracy. We demonstrate the result of our method on two datasets: Cityscapes and Mapillary Vistas. For Cityscapes, which has a large number of weakly labelled images, we also leverage auto-labelling to improve generalization. Using our approach we achieve a new state-of-the-art results in both Mapillary (61.1 IOU val) and Cityscapes (85.1 IOU test).

Motivation & Objective

• Motivate and address the trade-off between detail and global context in semantic segmentation across scales.
• Develop a memory-efficient attention mechanism that learns to weight adjacent scales at a pixel level.
• Enable flexible inference with variable scales without retraining.
• Improve generalization on Cityscapes via auto-labelling of coarse images.
• Demonstrate state-of-the-art performance on Cityscapes and Mapillary Vistas.

Proposed method

• Introduce a hierarchical attention mechanism that predicts relative attention between adjacent scales rather than full scale-wise attention masks.
• Train with adjacent scale pairs (e.g., r=1.0 and r=0.5) and chain attention during inference to combine N scales.
• Use a shared network trunk plus separate semantic and attention heads; apply attention masks through pixel-wise multiplication and addition to fuse multi-scale predictions.
• Adopt hard auto-labelling for coarse Cityscapes images to generate dense, efficient labels for improved generalization.
• Use DelOp DeepLab V3+ style backbone (ResNet-50 or HRNet-OCR) and train with random scaling augmentations and class-balanced sampling.
• Evaluate on Cityscapes and Mapillary Vistas to compare with averaging and explicit attention baselines.

Experimental results

Research questions

• RQ1 Can hierarchical adjacent-scale attention effectively replace full multi-scale attention masks while maintaining or improving segmentation accuracy?
• RQ2 Does enabling flexible inference with unseen scales (beyond training scales) improve performance without retraining?
• RQ3 What is the impact of hard auto-labelling on Cityscapes generalization and IoU scores?
• RQ4 How does memory and training efficiency of hierarchical attention compare to explicit multi-scale attention methods?
• RQ5 What is the performance gain on Cityscapes and Mapillary Vistas when combining hierarchical attention with auto-labelling?

Key findings

• Hierarchical multi-scale attention achieves higher IoU than both single-scale and average-pooling baselines on Mapillary (51.6) and Cityscapes (85.1 test) datasets.
• Adding a 0.25x scale with hierarchical attention improves Mapillary IoU by 0.6 and Cityscapes by enabling finer details without needing retraining.
• The hierarchical method is more memory-efficient, requiring 1.25x training FLOPs relative to single-scale, and supports flexible inference with additional scales.
• Hard auto-labelling of coarse Cityscapes images boosts Cityscapes IoU by about 1.1 percentage points over baseline, contributing to a total gain when combined with hierarchical attention.
• The approach achieves state-of-the-art results on Cityscapes (85.1 IoU) and Mapillary Vistas (61.1 IoU) at validation/test, respectively.
• Ablation indicates multi-scale attention yields gains over standard HRNet-OCR baselines and that the combination of MS Attention and auto-labelling provides the best results.