[Paper Review] Efficient piecewise training of deep structured models for semantic segmentation

Guosheng Lin, Chunhua Shen+2 more|arXiv (Cornell University)|Apr 4, 2015

Advanced Neural Network Applications42 references83 citations

TL;DR

This paper proposes a deep structured model that combines CNNs with Conditional Random Fields (CRFs) to improve semantic segmentation by modeling complex spatial context. It introduces CNN-based pairwise potentials for patch-patch context and uses piecewise training to enable efficient end-to-end learning, achieving a new state-of-the-art 78.0 mIoU on PASCAL VOC 2012 with multi-scale input and sliding pyramid pooling for background context.

ABSTRACT

Recent advances in semantic image segmentation have mostly been achieved by training deep convolutional neural networks (CNNs). We show how to improve semantic segmentation through the use of contextual information; specifically, we explore `patch-patch' context between image regions, and `patch-background' context. For learning from the patch-patch context, we formulate Conditional Random Fields (CRFs) with CNN-based pairwise potential functions to capture semantic correlations between neighboring patches. Efficient piecewise training of the proposed deep structured model is then applied to avoid repeated expensive CRF inference for back propagation. For capturing the patch-background context, we show that a network design with traditional multi-scale image input and sliding pyramid pooling is effective for improving performance. Our experimental results set new state-of-the-art performance on a number of popular semantic segmentation datasets, including NYUDv2, PASCAL VOC 2012, PASCAL-Context, and SIFT-flow. In particular, we achieve an intersection-over-union score of 78.0 on the challenging PASCAL VOC 2012 dataset.

Motivation & Objective

• To improve semantic segmentation by explicitly modeling contextual relationships between image regions and with the background.
• To address the computational inefficiency of joint training in CRF-based models with deep features.
• To enhance performance through structured modeling of patch-patch and patch-background context using deep neural networks.
• To achieve state-of-the-art results on benchmark datasets including PASCAL VOC 2012, NYUDv2, PASCAL-Context, and SIFT-flow.

Proposed method

• Formulates CNN-based pairwise potential functions in CRFs to model semantic compatibility between neighboring image patches.
• Applies piecewise training of CRFs to avoid repeated expensive inference during backpropagation, enabling efficient end-to-end learning.
• Uses multi-scale image input and sliding pyramid pooling to capture rich patch-background context.
• Employs refinement modules on intermediate feature maps to improve boundary details and resolution.
• Performs post-processing with boundary refinement to produce final high-resolution predictions.
• Combines coarse CRF-inferred predictions with refined features from multiple network layers for improved accuracy.

Experimental results

Research questions

• RQ1 Can CNN-based pairwise potentials in CRFs effectively model semantic compatibility between image patches for improved segmentation?
• RQ2 Can piecewise training of CRFs enable efficient joint learning of deep structured models without repeated inference?
• RQ3 How does incorporating multi-scale and pyramid-pooled features enhance background context modeling in semantic segmentation?
• RQ4 To what extent does the proposed method improve performance on challenging benchmarks like PASCAL VOC 2012?

Key findings

• The method achieves a new state-of-the-art mean intersection-over-union (mIoU) score of 78.0 on the PASCAL VOC 2012 dataset.
• With only VOC 2012 training data, the model achieves 75.3 mIoU, outperforming all prior methods using the same setting.
• Using additional COCO data, the mIoU increases to 77.2, demonstrating the effectiveness of large-scale pretraining.
• The method achieves 43.3 mIoU on PASCAL-Context (60 classes), the best reported result on this dataset.
• On the SIFT-flow dataset (33 classes), the method achieves 44.9 mIoU, outperforming all prior methods.
• The model outperforms the second-best method on 18 out of 20 categories when trained only on VOC 2012 data.