[Paper Review] Pyramid Scene Parsing Network

Hengshuang Zhao, Jianping Shi+3 more|arXiv (Cornell University)|Dec 4, 2016

Advanced Image and Video Retrieval Techniques36 references314 citations

TL;DR

PSPNet introduces a four-level pyramid pooling module to capture multi-scale global context for pixel-level scene parsing, achieving state-of-the-art results on ADE20K, PASCAL VOC 2012, and Cityscapes with a deeply supervised ResNet-based FCN.

ABSTRACT

Scene parsing is challenging for unrestricted open vocabulary and diverse scenes. In this paper, we exploit the capability of global context information by different-region-based context aggregation through our pyramid pooling module together with the proposed pyramid scene parsing network (PSPNet). Our global prior representation is effective to produce good quality results on the scene parsing task, while PSPNet provides a superior framework for pixel-level prediction tasks. The proposed approach achieves state-of-the-art performance on various datasets. It came first in ImageNet scene parsing challenge 2016, PASCAL VOC 2012 benchmark and Cityscapes benchmark. A single PSPNet yields new record of mIoU accuracy 85.4% on PASCAL VOC 2012 and accuracy 80.2% on Cityscapes.

Motivation & Objective

• Motivate accurate scene parsing under open vocabulary and diverse scenes by leveraging global scene context.
• Develop a pyramid pooling module to aggregate multi-scale contextual information.
• Integrate deep supervision to ease optimization of deep ResNet-based FCNs.
• Provide a practical, detailed implementation for reproducible scene parsing results.

Proposed method

• Propose a pyramid pooling module that fuses features from four pyramid levels (1x1, 2x2, 3x3, 6x6) on top of a dilated ResNet feature map.
• Concatenate pyramid-pooled features with the original feature map and apply a final convolution to predict per-pixel labels.
• Use average pooling (empirically found to be superior to max pooling) and apply 1x1 convolutions to reduce dimensionality before upsampling and concatenation.
• Adopt a deeply supervised training strategy by adding an auxiliary loss at an intermediate ResNet block (res4b22) to facilitate optimization of very deep networks.
• Train end-to-end with data augmentation and a poly learning rate policy; evaluate on ADE20K, PASCAL VOC 2012, and Cityscapes.

Experimental results

Research questions

• RQ1 Can a pyramid-based global context prior improve pixel-level scene parsing for open vocabulary datasets (e.g., ADE20K)?
• RQ2 Does multi-scale regional pooling better encode contextual relationships than global pooling alone?
• RQ3 Does deep supervision help train very deep ResNet-based FCNs for semantic segmentation?
• RQ4 What is the impact of pooling type (average vs. max) and dimensionality reduction on segmentation performance?

Key findings

• PSPNet with four-level pyramid pooling significantly improves Mean IoU and Pixel Accuracy over the ResNet-based baseline.
• Average pooling outperforms max pooling across configurations.
• Dimensionality reduction after pooling before concatenation yields better performance.
• Deeper networks (ResNet269) with auxiliary loss and PSP achieve higher accuracy, with multi-scale testing further boosting results.
• On ADE20K, PSPNet with deep supervision and PSP achieves substantial gains over baselines, and single-model results approach or surpass several multi-model ensembles in VOC 2012.
• On VOC 2012, PSPNet achieves 85.4% VOC accuracy with MS-COCO pre-training and strong per-class performance.