[Paper Review] Learning to Segment Object Candidates

Pedro O. Pinheiro, Ronan Collobert+1 more|arXiv (Cornell University)|Jun 20, 2015

Advanced Neural Network Applications33 references522 citations

TL;DR

This paper proposes DeepMask, a convolutional neural network that generates class-agnostic segmentation masks and object likelihood scores directly from raw image pixels, without relying on edges or superpixels. Trained jointly on MS COCO and evaluated on PASCAL VOC and COCO, it achieves state-of-the-art object proposal performance, significantly outperforming prior methods in recall with far fewer proposals—e.g., 68.2% mAP with 100 DeepMask proposals versus 66.9% with 2000 SelectiveSearch proposals.

ABSTRACT

Recent object detection systems rely on two critical steps: (1) a set of object proposals is predicted as efficiently as possible, and (2) this set of candidate proposals is then passed to an object classifier. Such approaches have been shown they can be fast, while achieving the state of the art in detection performance. In this paper, we propose a new way to generate object proposals, introducing an approach based on a discriminative convolutional network. Our model is trained jointly with two objectives: given an image patch, the first part of the system outputs a class-agnostic segmentation mask, while the second part of the system outputs the likelihood of the patch being centered on a full object. At test time, the model is efficiently applied on the whole test image and generates a set of segmentation masks, each of them being assigned with a corresponding object likelihood score. We show that our model yields significant improvements over state-of-the-art object proposal algorithms. In particular, compared to previous approaches, our model obtains substantially higher object recall using fewer proposals. We also show that our model is able to generalize to unseen categories it has not seen during training. Unlike all previous approaches for generating object masks, we do not rely on edges, superpixels, or any other form of low-level segmentation.

Motivation & Objective

• To develop a method for generating object proposals that surpasses existing approaches in recall and efficiency.
• To eliminate reliance on low-level cues such as edges, superpixels, or handcrafted features in proposal generation.
• To train a unified convolutional network that jointly predicts segmentation masks and object likelihood scores.
• To evaluate generalization to unseen object categories not present during training.
• To demonstrate improved detection performance when integrated with Fast R-CNN using fewer proposals.

Proposed method

• A shared convolutional backbone processes image patches and outputs features for both segmentation and objectness prediction.
• The segmentation branch uses a low-rank fully connected layer to predict a 56×56 class-agnostic mask from features.
• The objectness branch predicts the likelihood of a patch containing a complete object using a separate, discriminatively trained head.
• The model is trained end-to-end with a joint loss combining mask and score prediction objectives.
• At test time, the network is applied densely across the image at multiple scales to generate ranked segmentation proposals.
• Inference is accelerated via batch processing across scales and GPU acceleration.

Experimental results

Research questions

• RQ1 Can a deep convolutional network learn to generate high-quality object proposals directly from raw image pixels without relying on low-level segmentation?
• RQ2 Does joint training of segmentation and objectness prediction improve proposal quality compared to separate optimization?
• RQ3 Can the model generalize to object categories unseen during training, particularly in the segmentation branch?
• RQ4 Does using fewer, higher-quality proposals lead to better downstream detection performance?
• RQ5 How does the model perform across different object sizes and IoU thresholds?

Key findings

• DeepMask achieves 69.9% mAP on PASCAL VOC 2007 using only 500 proposals, outperforming Fast R-CNN with 2000 SelectiveSearch proposals (66.9% mAP).
• With 100 proposals, DeepMask achieves 68.2% mAP, surpassing the 66.9% mAP obtained with 2000 SelectiveSearch proposals.
• On PASCAL VOC 2007, DeepMask achieves 69.0% average recall at 1000 proposals (AR@1000), outperforming MCG (63.4%) and SelectiveSearch (61.8%).
• The model generalizes well: DeepMask20∗, trained on only 20 PASCAL categories, matches the performance of the full DeepMask model on 80 COCO categories.
• DeepMask achieves higher localization recall than all baselines at IoU thresholds below 0.7, except at very high IoU (≥0.9), where it slightly underperforms due to downsampled mask outputs.
• Inference takes 1.6s per image on COCO (1.2s on PASCAL), which is competitive with fast methods like Geodesic (~1s) and significantly faster than MCG (~30s).