[Paper Review] Simultaneous Detection and Segmentation

Bharath Hariharan, Pablo Arbeláez+2 more|arXiv (Cornell University)|Jul 7, 2014

Advanced Neural Network Applications26 references199 citations

TL;DR

This paper proposes a novel deep learning framework for Simultaneous Detection and Segmentation (SDS), combining region proposals with category-specific, top-down refinement using convolutional neural networks. The method achieves state-of-the-art performance with a 7-point APr boost (16% relative) over baselines, 5-point improvement in semantic segmentation (10% relative), and improved object detection, demonstrating the effectiveness of joint training for segmentation and detection.

ABSTRACT

We aim to detect all instances of a category in an image and, for each instance, mark the pixels that belong to it. We call this task Simultaneous Detection and Segmentation (SDS). Unlike classical bounding box detection, SDS requires a segmentation and not just a box. Unlike classical semantic segmentation, we require individual object instances. We build on recent work that uses convolutional neural networks to classify category-independent region proposals (R-CNN [16]), introducing a novel architecture tailored for SDS. We then use category-specific, top- down figure-ground predictions to refine our bottom-up proposals. We show a 7 point boost (16% relative) over our baselines on SDS, a 5 point boost (10% relative) over state-of-the-art on semantic segmentation, and state-of-the-art performance in object detection. Finally, we provide diagnostic tools that unpack performance and provide directions for future work.

Motivation & Objective

• Address the gap between object detection (bounding boxes) and semantic segmentation (pixel-level labels) by unifying both tasks into a single framework.
• Develop a method that detects all instances of a category and assigns precise pixel-level segmentation masks to each.
• Improve performance on both detection and segmentation tasks by jointly training feature extractors for region proposals and foreground masks.
• Introduce new evaluation metrics—APr and APvol—to better assess segmentation accuracy and instance-level localization.
• Diagnose failure modes in detection and segmentation to guide future model improvements.

Proposed method

• Generate 2000 category-independent region proposals per image using MCG (Maximal Stable Static Regions).
• Extract features from both the bounding box and the region mask using separate, finetuned CNNs, with joint end-to-end training for improved feature learning.
• Classify each region proposal using an SVM trained on CNN features to predict category scores.
• Apply non-maximum suppression (NMS) on scored proposals and refine surviving candidates using category-specific coarse mask predictions.
• Combine refined masks with original region proposals to improve localization and segmentation accuracy.
• Convert final outputs to pixel-level labels using a pasting scheme for evaluation on semantic segmentation benchmarks.

Experimental results

Research questions

• RQ1 Can a unified deep learning framework jointly optimize for object detection and instance-level segmentation with better performance than separate task models?
• RQ2 How does joint training of region proposal and foreground mask features impact segmentation and detection accuracy?
• RQ3 What is the impact of mislocalization and false positives on segmentation performance, and how can they be diagnosed and mitigated?
• RQ4 To what extent does the proposed SDS framework improve state-of-the-art results in semantic segmentation and object detection?
• RQ5 How do varying overlap thresholds affect performance, and what metrics best capture the full performance surface across thresholds?

Key findings

• The proposed SDS framework achieves a 49.5% mean APr, representing a 7-point absolute (16% relative) improvement over baseline methods.
• The method improves mean APb from 51.0% (R-CNN) to 53.0%, demonstrating state-of-the-art performance in object detection.
• On semantic segmentation, the method achieves 52.6% mean pixel IU on VOC2011 test and 51.6% on VOC2012 test, a 5-point absolute (10% relative) improvement over the prior state-of-the-art.
• The APvol metric shows a 41.4% value for SDS, indicating strong performance across varying overlap thresholds, with a significant reduction in mislocalization errors.
• Diagnostic analysis reveals that mislocalization is the largest contributor to performance loss (15.8% APr loss in best model), and that refining proposals with category-specific masks reduces this error.
• The model generalizes well to complex scenes, correctly resolving individual instances even in cluttered or unusual poses, as shown in qualitative examples.