[Paper Review] Shape-aware Feature Extraction for Instance Segmentation.

Hao Ding, Siyuan Qiao+2 more|arXiv (Cornell University)|Nov 25, 2019

Advanced Neural Network Applications35 references4 citations

TL;DR

This paper proposes Shape-aware RoIAlign, a novel feature extraction strategy that refines RoI features by aligning them to the predicted mask shape rather than a rectangular bounding box, improving instance segmentation in crowded scenes. The method significantly boosts performance on COCO, outperforming standard Mask R-CNN across multiple backbones.

ABSTRACT

Modern instance segmentation approaches mainly adopt a sequential paradigm - ``detect then segment'', as popularized by Mask R-CNN, which have achieved considerable progress. However, they usually struggle to segment huddled instances, i.e., instances which are crowded together. The essential reason is the detection step is only learned under box-level supervision. Without the guidance from the mask-level supervision, the features extracted from the regions containing huddled instances are noisy and ambiguous, which makes the detection problem ill-posed. To address this issue, we propose a new region-of-interest (RoI) feature extraction strategy, named Shape-aware RoIAlign, which focuses feature extraction within a region aligned well with the shape of the instance-of-interest rather than a rectangular RoI. We instantiate Shape-aware RoIAlign by introducing a novel refining module built upon Mask R-CNN, which takes the mask predicted by Mask R-CNN as the region to guide the computation of Shape-aware RoIAlign. Based on the RoI features re-computed by Shape-aware RoIAlign, the refining module updates the bounding box as well as the mask predicted by Mask R-CNN. Experimental results show that the refining module equipped with Shape-aware RoIAlign achieves consistent and remarkable improvements than Mask R-CNN models with different backbones, respectively, on the challenging COCO dataset. The code will be released.

Motivation & Objective

• Address the challenge of segmenting huddled instances in instance segmentation, where overlapping objects degrade performance.
• Overcome the limitation of box-level supervision in detection heads, which leads to noisy and ambiguous features in crowded regions.
• Improve feature quality in regions with overlapping instances by leveraging mask-level supervision during RoI feature extraction.
• Develop a refining module that uses predicted masks to guide spatial alignment and re-compute RoI features for better bounding box and mask predictions.

Proposed method

• Propose Shape-aware RoIAlign, a modified RoI alignment strategy that uses the predicted mask of an instance as a shape-aware region of interest instead of a fixed rectangular RoI.
• Design a refining module that takes the initial mask prediction from Mask R-CNN as a spatial guide to re-compute RoI features with Shape-aware RoIAlign.
• Update both the bounding box and mask predictions using the refined RoI features, improving localization and segmentation accuracy.
• Integrate the refining module into the Mask R-CNN framework with end-to-end training, enabling the model to learn from mask-level supervision during feature extraction.
• Use the predicted mask as a spatial mask to restrict feature sampling to the actual instance shape, reducing interference from neighboring objects.

Experimental results

Research questions

• RQ1 Can mask-level supervision improve feature quality in crowded regions where box-level supervision fails?
• RQ2 Does aligning RoI features to the predicted instance shape rather than a rectangular box lead to better segmentation performance?
• RQ3 Can a refining module based on Shape-aware RoIAlign consistently improve performance across different backbone architectures on instance segmentation benchmarks?
• RQ4 How does Shape-aware RoIAlign compare to standard RoIAlign in handling huddled or overlapping instances?

Key findings

• Shape-aware RoIAlign achieves consistent and significant improvements over standard Mask R-CNN across multiple backbone architectures on the COCO dataset.
• The refining module with Shape-aware RoIAlign enhances both bounding box and mask prediction quality, especially in crowded scenes with overlapping instances.
• The method effectively reduces noisy and ambiguous features in overlapping regions by focusing on the actual instance shape during feature extraction.
• The performance gains are quantitatively measurable, with reported improvements on COCO, though exact mAP values are not specified in the provided text.