[Paper Review] Bridging Category-level and Instance-level Semantic Image Segmentation

Zifeng Wu, Chunhua Shen+1 more|ArXiv.org|May 23, 2016

Advanced Neural Network ApplicationsComputer Science24 references158 citations

TL;DR

The paper builds an instance segmentation approach atop category-level semantic segmentation by predicting instance bounding-box transforms from semantic masks, introduces online bootstrapping for hard pixels, and achieves state-of-the-art semantic IoU and competitive instance segmentation on PASCAL VOC 2012.

ABSTRACT

We propose an approach to instance-level image segmentation that is built on top of category-level segmentation. Specifically, for each pixel in a semantic category mask, its corresponding instance bounding box is predicted using a deep fully convolutional regression network. Thus it follows a different pipeline to the popular detect-then-segment approaches that first predict instances' bounding boxes, which are the current state-of-the-art in instance segmentation. We show that, by leveraging the strength of our state-of-the-art semantic segmentation models, the proposed method can achieve comparable or even better results to detect-then-segment approaches. We make the following contributions. (i) First, we propose a simple yet effective approach to semantic instance segmentation. (ii) Second, we propose an online bootstrapping method during training, which is critically important for achieving good performance for both semantic category segmentation and instance-level segmentation. (iii) As the performance of semantic category segmentation has a significant impact on the instance-level segmentation, which is the second step of our approach, we train fully convolutional residual networks to achieve the best semantic category segmentation accuracy. On the PASCAL VOC 2012 dataset, we obtain the currently best mean intersection-over-union score of 79.1%. (iv) We also achieve state-of-the-art results for instance-level segmentation.

Motivation & Objective

• Motivate and develop an instance segmentation method that leverages strong category-level segmentation
• Transform semantic score maps into candidate instance locations via a regression-based bounding-box transform
• Improve training with online bootstrapping of hard pixels to enhance both semantic and instance segmentation
• Design and evaluate a fully convolutional residual network (FCRN) with large field-of-view and dilated convolutions
• Demonstrate state-of-the-art semantic segmentation and competitive instance segmentation on standard benchmarks (PASCAL VOC 2012, Cityscapes, PASCAL-Context)

Proposed method

• Predict category-wise semantic score maps using an FCN-based residual network (FCRN)
• Learn a localization/regression network to predict vertical/horizontal offsets and height/width of instance bounding boxes from each pixel
• Transform semantic score maps with the predicted bounding-box transforms to obtain per-pixel localization maps
• Apply non-maximum suppression on transformed maps to generate instance hypotheses
• Trace back and average scores within NMS-derived regions to form instance masks, followed by region-based NMS for final results
• Train with online bootstrapping that down-weights easy pixels (semantic: based on p_ij < t; localization: based on IoU between predicted and ground-truth boxes) to focus on hard examples
• Construct FCRNs by adapting ResNet into fully-convolutional networks, removing the final pooling layer, employing atrous/dilated convolutions to achieve large FoV while preserving resolution, and enabling high-resolution feature maps for pixel-wise classification

Experimental results

Research questions

• RQ1 Can high-quality instance segmentation be achieved by building on top of semantic category-level segmentation without bounding-box detectors?
• RQ2 Does online bootstrapping of hard pixels improve both semantic and instance segmentation performance?
• RQ3 How do architectural choices (depth, resolution, field-of-view, dilation) in fully convolutional residual networks affect segmentation accuracy?
• RQ4 What is the impact of improved semantic segmentation on downstream instance segmentation performance?
• RQ5 How does the proposed method compare to bounding-box detection–based instance segmentation on standard benchmarks?

Key findings

• Semantic segmentation achieves state-of-the-art mean IoU on PASCAL VOC 2012 with 79.1% using augmented training data
• Instance-level results are on par or better than previous best methods on PASCAL VOC 2012, with a notable 5.1% absolute improvement in mean AP for mAP_r at IoU 0.7 (from 41.5% to 46.6%)
• Online bootstrapping of hard pixels significantly improves performance, including a 3.1% IoU gain on Cityscapes
• Best-performing semantic model (FCRN with bootstrapping) yields strong semantic accuracy, which in turn enhances instance segmentation performance
• Pre-training the semantic network on COCO can further improve instance segmentation performance by about 2.0% (mAP_r at IoU 0.5)
• Qualitative results illustrate accurate semantic delineation and coherent instance masks without relying on bounding-box detectors