[Paper Review] DeeperLab: Single-Shot Image Parser

Tien-Ju Yang, Maxwell D. Collins+7 more|arXiv (Cornell University)|Feb 13, 2019

Advanced Neural Network ApplicationsComputer Science101 references162 citations

TL;DR

DeeperLab presents a single-shot, bottom-up image parser for panoptic segmentation that jointly predicts semantic and instance labels in one pass, with a region-aware Parsing Covering metric and strong Mapillary Vistas results.

ABSTRACT

We present a single-shot, bottom-up approach for whole image parsing. Whole image parsing, also known as Panoptic Segmentation, generalizes the tasks of semantic segmentation for 'stuff' classes and instance segmentation for 'thing' classes, assigning both semantic and instance labels to every pixel in an image. Recent approaches to whole image parsing typically employ separate standalone modules for the constituent semantic and instance segmentation tasks and require multiple passes of inference. Instead, the proposed DeeperLab image parser performs whole image parsing with a significantly simpler, fully convolutional approach that jointly addresses the semantic and instance segmentation tasks in a single-shot manner, resulting in a streamlined system that better lends itself to fast processing. For quantitative evaluation, we use both the instance-based Panoptic Quality (PQ) metric and the proposed region-based Parsing Covering (PC) metric, which better captures the image parsing quality on 'stuff' classes and larger object instances. We report experimental results on the challenging Mapillary Vistas dataset, in which our single model achieves 31.95% (val) / 31.6% PQ (test) and 55.26% PC (val) with 3 frames per second (fps) on GPU or near real-time speed (22.6 fps on GPU) with reduced accuracy.

Motivation & Objective

• Motivate efficient whole image parsing (panoptic segmentation) that unifies semantic and instance segmentation.
• Propose a single-pass, bottom-up architecture that shares features between semantic and instance heads for speed and simplicity.
• Introduce design strategies to reduce memory usage and improve receptive field for high-resolution inputs.
• Propose the Parsing Covering (PC) metric as a region-based alternative to Panoptic Quality (PQ).
• Demonstrate state-of-the-art performance-speed trade-offs on the Mapillary Vistas dataset.

Proposed method

• Use a shared encoder-decoder backbone with depthwise separable convolutions and a strong receptive field via larger kernels and ASPP.
• Predict per-pixel semantic labels and class-agnostic instance cues through five heads attached to a common decoder output.
• Employ space-to-depth and depth-to-space operations to manage memory during upsampling.
• Use hard pixel mining and weighted loss to emphasize small instances and hard pixels during training.
• Fuse semantic and instance predictions in a simple, fast final fusion step to produce panoptic parsing results.

Experimental results

Research questions

• RQ1 Can a single-shot, bottom-up network jointly produce semantic and instance segmentation for full-image parsing efficiently?
• RQ2 How do architectural choices (kernel size, decoder design, ASPP, memory management) affect accuracy and speed on high-resolution street-scene data?
• RQ3 Does a region-based metric like Parsing Covering provide complementary insights to Panoptic Quality for image parsing?
• RQ4 What is the trade-off between accuracy and throughput for different backbones (Xception-71 vs Wider MobileNetV2) on Mapillary Vistas?

Key findings

• Xception-71 backbone achieves 31.95% PQ and 55.26% PC on Mapillary Vistas validation set at 3.09 fps on GPU.
• Wider MobileNetV2 with ASPP reaches 25.20% PQ and 49.80% PC at 6.19 fps on GPU, with faster overall speed.
• Light Wider MobileNetV2 downsampled input yields near real-time speed (22.61 fps on GPU) with some accuracy loss.
• Decoder design with space-to-depth and depth-to-space, larger kernels, and deeper prediction heads improves PQ/PC substantially over baselines.
• Hard pixel mining and increased small-instance loss weighting yield measurable PQ/PC gains (e.g., PQ up to 24.99%, PC up to 49.23%).
• The paper also proposes Parsing Covering (PC) as a region-based parsing quality metric and reports results on Cityscapes, Pascal VOC 2012, and COCO in supplementary material.