[Paper Review] Deep GrabCut for Object Selection

Ning Xu, Brian Price+3 more|arXiv (Cornell University)|Jul 2, 2017

Advanced Neural Network Applications17 references14 citations

TL;DR

This paper proposes Deep GrabCut, a deep learning method that transforms input rectangles into Euclidean distance maps to improve object segmentation robustness. By fusing images with distance maps in a U-Net style encoder-decoder network, the model achieves accurate, context-aware segmentation even with poorly placed bounding boxes, generalizes to arbitrary closed curves without retraining, and enhances instance-level semantic segmentation via CRF-based labeling.

ABSTRACT

Most previous bounding-box-based segmentation methods assume the bounding box tightly covers the object of interest. However it is common that a rectangle input could be too large or too small. In this paper, we propose a novel segmentation approach that uses a rectangle as a soft constraint by transforming it into an Euclidean distance map. A convolutional encoder-decoder network is trained end-to-end by concatenating images with these distance maps as inputs and predicting the object masks as outputs. Our approach gets accurate segmentation results given sloppy rectangles while being general for both interactive segmentation and instance segmentation. We show our network extends to curve-based input without retraining. We further apply our network to instance-level semantic segmentation and resolve any overlap using a conditional random field. Experiments on benchmark datasets demonstrate the effectiveness of the proposed approaches.

Motivation & Objective

• To address the limitation of bounding-box-based segmentation methods that fail when rectangles are poorly placed or too loose.
• To improve segmentation accuracy by incorporating global context and spatial relationships beyond local color and edge features.
• To enable robust, general-purpose object selection using loose or imprecise rectangles as input.
• To extend the method to arbitrary closed curves without retraining, enabling more flexible user interaction.
• To enhance instance-level semantic segmentation by resolving overlaps through CRF-based labeling.

Proposed method

• Transforms input rectangles into Euclidean distance maps to encode spatial relationships and object proximity.
• Concatenates the RGB image and distance map as input to a U-Net style encoder-decoder network (CEDN) for end-to-end training.
• Trains the network on loosely placed rectangles to improve robustness to poor bounding box placement.
• Uses a conditional random field (CRF) to resolve overlapping detections and produce unique instance-level labels.
• Applies the same trained network to curve-based inputs by transforming curves into distance maps using the same distance computation.
• Leverages the model's understanding of global context to maintain accuracy even when the input shape is not a tight bounding box.

Experimental results

Research questions

• RQ1 Can a deep learning model achieve accurate object segmentation when given loose or poorly placed bounding boxes as input?
• RQ2 How does incorporating a distance map representation improve segmentation robustness compared to traditional bounding box methods?
• RQ3 Can a model trained exclusively on rectangular inputs generalize effectively to arbitrary closed curves like circles or freeform shapes?
• RQ4 To what extent does the model’s use of global context improve segmentation accuracy compared to local color and edge models?
• RQ5 How effective is the CRF-based labeling approach in resolving overlapping detections for instance-level semantic segmentation?

Key findings

• The proposed method achieves higher mean Intersection over Union (mIoU) than baseline methods across all rectangle IOU bins, with mask IoU consistently exceeding detection IoU (e.g., 0.7 vs. 0.6 at 0.6 detection IoU).
• The model shows significantly less performance degradation under rectangle jittering (up to ±128 pixels) compared to baseline methods, demonstrating robustness to input placement.
• The model generalizes effectively to arbitrary closed curves without retraining, enabling accurate selection of partial objects (e.g., one half of a stone) that bounding boxes cannot isolate.
• The CRF-based labeling approach successfully resolves overlapping detections and preserves labeling consistency, outperforming naive score-thresholding baselines in complex scenes.
• Visual results show that the method produces accurate, coherent masks even when rectangles are too small or too large, and handles challenging cases like internal boxes or full-image boxes.
• The method maintains high performance on benchmark datasets, demonstrating strong generalization across diverse object shapes and contexts.