[Paper Review] SPGNet: Semantic Prediction Guidance for Scene Parsing

Bowen Cheng, Liang-Chieh Chen+7 more|arXiv (Cornell University)|Aug 26, 2019

Human Pose and Action RecognitionComputer Science78 references35 citations

TL;DR

SPGNet introduces a Semantic Prediction Guidance (SPG) module within a two-stage encoder-decoder network to re-weight local features via pixel-wise semantic supervision, achieving strong Cityscapes results with high efficiency.

ABSTRACT

Multi-scale context module and single-stage encoder-decoder structure are commonly employed for semantic segmentation. The multi-scale context module refers to the operations to aggregate feature responses from a large spatial extent, while the single-stage encoder-decoder structure encodes the high-level semantic information in the encoder path and recovers the boundary information in the decoder path. In contrast, multi-stage encoder-decoder networks have been widely used in human pose estimation and show superior performance than their single-stage counterpart. However, few efforts have been attempted to bring this effective design to semantic segmentation. In this work, we propose a Semantic Prediction Guidance (SPG) module which learns to re-weight the local features through the guidance from pixel-wise semantic prediction. We find that by carefully re-weighting features across stages, a two-stage encoder-decoder network coupled with our proposed SPG module can significantly outperform its one-stage counterpart with similar parameters and computations. Finally, we report experimental results on the semantic segmentation benchmark Cityscapes, in which our SPGNet attains 81.1% on the test set using only 'fine' annotations.

Motivation & Objective

• Motivate semantic segmentation with efficient multi-stage architectures.
• Propose SPG module to re-weight features guided by pixel-level semantic predictions.
• Explore multi-stage encoder-decoder networks for improved boundary and context fusion.
• Evaluate on Cityscapes to demonstrate accuracy and efficiency gains.
• Provide ablations and visualizations to explain the SPG mechanism.

Proposed method

• Introduce SPG module that uses a supervise-and-excite framework to generate Guided Attention from first-stage predictions.
• Use a two-stage encoder-decoder with Cross Stage Feature Aggregation to enhance later stages.
• Design a lightweight upsample module with residual blocks for efficient feature fusion.
• Compute Guided Attention via a 1x1 conv producing per-pixel per-channel masks, and re-weight decoder features accordingly.
• Train with losses from the final stage and intermediate stage logits for supervision.
• Compare against state-of-the-art on Cityscapes and conduct extensive ablations and visualizations.

Experimental results

Research questions

• RQ1 Does the SPG module improve feature re-weighting and segmentation accuracy when guided by pixel-wise semantic predictions?
• RQ2 Can a two-stage encoder-decoder with SPG outperform a single-stage counterpart with similar parameters and compute?
• RQ3 How does SPGNet compare to DenseASPP and DANet on Cityscapes in terms of accuracy and efficiency?
• RQ4 What are the contributions of SPG components (supervision, identity mapping, excitation mechanism) to overall performance?
• RQ5 Is multi-stage encoder-decoder networking beneficial for semantic segmentation when coupled with SPG?

Key findings

• SPGNet attains 81.1% mean IoU on Cityscapes test using only fine annotations.
• SPGNet outperforms DenseASPP on Cityscapes test in most classes and uses about 22.7% of the computation of DANet.
• A 2-stage SPGNet with ResNet-50 backbones achieves strong accuracy with markedly lower FLOPs and parameter count compared to many top methods.
• Ablations show sigmoid-based SPG excitation with supervision and an identity path yields best mIoU (77.67% on val with ResNet-18).
• Guided Attention maps provide interpretable reweighting and visualize object localization and discrimination among similar classes.