[Paper Review] Semantic Instance Segmentation via Deep Metric Learning

Alireza Fathi, Zbigniew Wojna+5 more|arXiv (Cornell University)|Mar 30, 2017

Advanced Neural Network ApplicationsComputer Science25 references211 citations

TL;DR

The paper learns a pixel-level embedding space to measure whether two pixels belong to the same object and uses seed-based masking to produce instance segmentations, achieving competitive mAP on Pascal VOC 2012.

ABSTRACT

We propose a new method for semantic instance segmentation, by first computing how likely two pixels are to belong to the same object, and then by grouping similar pixels together. Our similarity metric is based on a deep, fully convolutional embedding model. Our grouping method is based on selecting all points that are sufficiently similar to a set of "seed points", chosen from a deep, fully convolutional scoring model. We show competitive results on the Pascal VOC instance segmentation benchmark.

Motivation & Objective

• Motivate semantic instance segmentation beyond bounding boxes by distinguishing instances of the same class.
• Propose a deep embedding model that encodes pixel-wise similarity to enable object-level grouping.
• Introduce a seed-based mask growth mechanism guided by learned seediness scores.
• Jointly train embedding and seediness/classification heads on a shared CNN backbone.

Proposed method

• Learn 64-dimensional pixel embeddings to make same-instance pixels close and different-instance pixels far in embedding space.
• Define pixel similarity as sigma(p,q)=2/(1+exp(||e_p-e_q||^2)) and train with a cross-entropy loss over pairs.
• Grow object masks by selecting seeds and expanding to all pixels with sigma(p,q) above a threshold tau.
• Compute a seediness heatmap to select high-quality seeds and promote spatial diversity using a max-diversity criterion.
• Attach class labels and confidence to each mask using a per-seed classification head trained with IoU-based guidance.
• Use a shared DeepLab v2 backbone with a multi-scale image pyramid and train embedding and seediness/classification heads jointly with controlled learning rates.

Experimental results

Research questions

• RQ1 Can a pixel-level embedding space effectively distinguish object instances across classes in a fully convolutional framework?
• RQ2 How can seed points be selected to efficiently generate high-quality instance masks without exhaustive pixel-wise grouping?
• RQ3 What is the impact of seediness-based seed selection and multi-threshold masking on segmentation accuracy?
• RQ4 Does a box-free, seed-driven approach compete with proposal-based instance segmentation methods on standard benchmarks?

Key findings

• Achieves mAP^r of 62.21% on Pascal VOC 2012 validation at IoU 0.5, ranking 4th among methods evaluated on VOC with proposals.
• Per-class results show strong performance on large objects (e.g., trains, dogs, motorbikes) but weaker on bicycles due to dataset annotation quality differences.
• Seed-based masking with top 100 seeds provides good coverage and recall without needing object proposals.
• The method performs better at higher IoU thresholds, ranking 2nd at IoU 0.6 and tied for 3rd at IoU 0.7 in the VOC 2012 evaluation.
• Embedding dimension of 64 with a seeds-based sampling parameter alpha around 0.3 yields strong mAP^r on the validation set.