[Paper Review] Pyramid Attention Network for Semantic Segmentation

Hanchao Li, Pengfei Xiong+2 more|arXiv (Cornell University)|May 25, 2018

Advanced Neural Network Applications26 references235 citations

TL;DR

PAN combines Feature Pyramid Attention and Global Attention Upsample to exploit global context and multi-scale features, achieving state-of-the-art mIoU on VOC 2012 (84.0%) and Cityscapes without COCO pretraining.

ABSTRACT

A Pyramid Attention Network(PAN) is proposed to exploit the impact of global contextual information in semantic segmentation. Different from most existing works, we combine attention mechanism and spatial pyramid to extract precise dense features for pixel labeling instead of complicated dilated convolution and artificially designed decoder networks. Specifically, we introduce a Feature Pyramid Attention module to perform spatial pyramid attention structure on high-level output and combining global pooling to learn a better feature representation, and a Global Attention Upsample module on each decoder layer to provide global context as a guidance of low-level features to select category localization details. The proposed approach achieves state-of-the-art performance on PASCAL VOC 2012 and Cityscapes benchmarks with a new record of mIoU accuracy 84.0% on PASCAL VOC 2012, while training without COCO dataset.

Motivation & Objective

• Motivate improved semantic segmentation by leveraging global contextual information without relying on dilated convolutions or complex decoders.
• Introduce a lightweight decoder that uses high-level context to guide low-level localization.
• Design and integrate a Feature Pyramid Attention module to fuse multi-scale context with pixel-level attention.
• Develop a Global Attention Upsample module to reconstruct high-resolution predictions with guidance from global context.
• Demonstrate state-of-the-art performance on VOC 2012 and Cityscapes without COCO pretraining.

Proposed method

• Introduce Feature Pyramid Attention (FPA) to fuse multi-scale context using a pyramid structure with 3x3, 5x5, 7x7 convolutions and a global pooling branch, multiplying attention with the original features to preserve localization.
• Propose Global Attention Upsample (GAU) as a decoder that uses global context from high-level features to weight low-level features before gradual upsampling.
• Use ResNet-101 with dilated convolution (rate 2 in res5b) as the encoder backbone.
• Replace the 7x7 ResNet-101 layer with three 3x3 convolutions to reduce parameters.
• Train with standard cross-entropy loss, SGD, and poly learning rate policy, with data augmentation (flip and scaling).
• Demonstrate that FPA can outperform PSPNet and DeepLabv3 under the same output stride, and that GAU improves localization when combined with FPA.

Experimental results

Research questions

• RQ1 Can a pyramid-attention based module provide pixel-level multi-scale context without heavy dilated convolutions or complex decoders?
• RQ2 Does a global-context guided upsampling decoder improve boundary localization with minimal computational cost?
• RQ3 How do FPA and GAU individually and jointly affect performance on standard benchmarks like VOC 2012 and Cityscapes?
• RQ4 What is the impact of training without COCO pretraining on VOC 2012 and Cityscapes using the PAN architecture?

Key findings

• Feature Pyramid Attention with average pooling and 3x3/5x5/7x7 kernels plus a global pooling branch yields substantial gains over a baseline ResNet-101, reaching 78.37–78.37% mean IoU on VOC 2012 validation depending on configuration.
• Global Attention Upsample, when combined with GAU, improves mean IoU on VOC 2012 val from 72.60% to 77.84% (and up to 78.37% with GAU+FPA variants).
• PAN without COCO pretraining achieves 84.0% mean IoU on VOC 2012 test set, outperforming several state-of-the-art methods listed (e.g., EncNet, PSPNet, DeepLabv3) under comparable settings.
• On Cityscapes, PAN achieves 78.6% mean IoU on the test set without coarse annotations, slightly surpassing several prior methods.
• Compared to methods that rely on COCO data for training (e.g., Global Convolution Network), PAN achieves competitive performance without COCO pretraining.
• Ablation studies show average pooling in FPA outperforms max pooling, and the inclusion of a global pooling branch and larger kernels (3x3, 5x5, 7x7) improves performance.