[Paper Review] Semantic Instance Segmentation with a Discriminative Loss Function

Bert De Brabandere, Davy Neven+1 more|arXiv (Cornell University)|Aug 8, 2017

Advanced Neural Network ApplicationsComputer Science38 references442 citations

TL;DR

It presents a pixel-level discriminative loss that maps pixels to embeddings so that same-instance pixels cluster together and different instances are separated, enabling simple post-processing for instance segmentation without proposals or recurrence.

ABSTRACT

Semantic instance segmentation remains a challenging task. In this work we propose to tackle the problem with a discriminative loss function, operating at the pixel level, that encourages a convolutional network to produce a representation of the image that can easily be clustered into instances with a simple post-processing step. The loss function encourages the network to map each pixel to a point in feature space so that pixels belonging to the same instance lie close together while different instances are separated by a wide margin. Our approach of combining an off-the-shelf network with a principled loss function inspired by a metric learning objective is conceptually simple and distinct from recent efforts in instance segmentation. In contrast to previous works, our method does not rely on object proposals or recurrent mechanisms. A key contribution of our work is to demonstrate that such a simple setup without bells and whistles is effective and can perform on par with more complex methods. Moreover, we show that it does not suffer from some of the limitations of the popular detect-and-segment approaches. We achieve competitive performance on the Cityscapes and CVPPP leaf segmentation benchmarks.

Motivation & Objective

• Motivate and address the challenge of semantic instance segmentation without object proposals or recurrent models.
• Introduce a discriminative, metric-learning–inspired loss that clusters pixel embeddings by instance.
• Show that a standard segmentation backbone can be re-used with the new loss for competitive performance.
• Demonstrate effectiveness on Cityscapes instance-level labeling and CVPPP leaf segmentation benchmarks.

Proposed method

• Map each pixel to an n-dimensional embedding via a segmentation network.
• Use a discriminative loss with three terms: intra-cluster variance (pulls embeddings to cluster centers), inter-cluster distance (pushes cluster centers apart), and a regularization term.
• Employ hinge-based (margin) losses for both variance and distance terms to allow local manifold representations.
• During inference, cluster embeddings by thresholding around cluster centers or via mean-shift-inspired refinement to obtain discrete instances.
• Train with an off-the-shelf architecture (ResNet-38) and adapt only the loss and small post-processing steps.
• Compare against proposal-based and recurrent methods to highlight simplicity and effectiveness.

Experimental results

Research questions

• RQ1 Can a pixel-level discriminative loss enable accurate instance segmentation without proposals or recurrent architectures?
• RQ2 How does clustering-based post-processing of learned embeddings fare on challenging benchmarks like Cityscapes and CVPPP?
• RQ3 What is the impact of semantic segmentation quality and clustering strategy on overall instance segmentation performance?
• RQ4 How does the proposed method compare to state-of-the-art methods in accuracy and complexity?

Key findings

• The discriminative loss yields competitive instance segmentation performance on Cityscapes and CVPPP, matching state-of-the-art non-proposal methods.
• On CVPPP, the approach achieves a SBD score of 84.2, comparable to the state-of-the-art 84.9 while using a simpler pipeline.
• On Cityscapes, the method achieves competitive AP metrics and demonstrates robustness to occlusions and complex scenes without relying on object proposals.
• Performance is strongly influenced by the quality of semantic segmentation and by the clustering strategy, with ground-truth components yielding substantial gains in ablations.
• The approach benefits from a simple post-processing step (thresholding/mean-shift) rather than heavy CRFs or recurrent decoders, enabling efficient inference.