[Paper Review] Joint Object and Part Segmentation using Deep Learned Potentials

Peng Wang, Xiaohui Shen+4 more|arXiv (Cornell University)|May 1, 2015

Advanced Neural Network Applications55 references27 citations

TL;DR

This paper proposes a joint deep learning framework for simultaneous semantic object and part segmentation using semantic compositional parts (SCP) and a fully connected CRF. By training a two-channel FCN to predict SCP and object potentials, and refining predictions with long-range context via FCRF, the method achieves state-of-the-art performance, improving mean IOU by over 5% on part segmentation and 5.3% on object segmentation compared to prior methods.

ABSTRACT

Segmenting semantic objects from images and parsing them into their respective semantic parts are fundamental steps towards detailed object understanding in computer vision. In this paper, we propose a joint solution that tackles semantic object and part segmentation simultaneously, in which higher object-level context is provided to guide part segmentation, and more detailed part-level localization is utilized to refine object segmentation. Specifically, we first introduce the concept of semantic compositional parts (SCP) in which similar semantic parts are grouped and shared among different objects. A two-channel fully convolutional network (FCN) is then trained to provide the SCP and object potentials at each pixel. At the same time, a compact set of segments can also be obtained from the SCP predictions of the network. Given the potentials and the generated segments, in order to explore long-range context, we finally construct an efficient fully connected conditional random field (FCRF) to jointly predict the final object and part labels. Extensive evaluation on three different datasets shows that our approach can mutually enhance the performance of object and part segmentation, and outperforms the current state-of-the-art on both tasks.

Motivation & Objective

• To address the mutual dependency between object and part segmentation by jointly modeling both tasks to improve accuracy.
• To reduce ambiguity in part labeling across similar object categories (e.g., horse vs. cow legs) through shared semantic compositional parts (SCP).
• To leverage long-range contextual relationships via a fully connected CRF to refine both object and part predictions.
• To overcome error propagation in sequential pipelines by training and inferring object and part segmentation in an end-to-end, consistent manner.

Proposed method

• Introduces semantic compositional parts (SCP) to group visually and structurally similar parts across different object classes (e.g., legs of horses and cows).
• Trains a two-channel fully convolutional network (FCN) to predict SCP potentials and object potentials at multiple image scales.
• Concatenates SCP and object potentials and passes them through an additional convolutional layer to refine joint object potentials.
• Generates compact region proposals from SCP predictions to serve as nodes in a fully connected CRF (FCRF).
• Uses the FCRF to jointly infer final object and part labels, enforcing consistency through long-range contextual constraints.
• Applies the FCRF to refine predictions by exploring long-range dependencies, improving boundary accuracy and reducing local ambiguities.

Experimental results

Research questions

• RQ1 Can joint learning of object and part segmentation improve performance on both tasks compared to sequential or independent approaches?
• RQ2 How can shared part representations (SCP) reduce ambiguity in part labeling across similar object classes?
• RQ3 To what extent does incorporating long-range context via a fully connected CRF enhance segmentation accuracy for both objects and parts?
• RQ4 Can end-to-end joint training and inference reduce error propagation from object to part segmentation?

Key findings

• The proposed method achieves a 78.25% mean IOU on object segmentation, a 5.3% improvement over the baseline FCN (72.99%).
• On semantic part segmentation, the method achieves a 48.16% mean IOU, a 5.05% improvement over the prior state-of-the-art HC method (43.11%).
• The full model with joint FCN and FCRF inference outperforms the variant without FCRF by over 4% on object segmentation, demonstrating the value of long-range context.
• The FCRF with joint potentials improves performance by 4% over the FCN baseline, showing that joint learning provides better evidence for graphical model inference.
• Qualitative results show the model successfully resolves local ambiguities—e.g., correctly identifying cow legs despite appearance similarity to horse legs—by leveraging object-scale context.
• The method outperforms sequential pipelines like HC, which suffer from error propagation due to inaccurate object masks affecting part labeling.