[Paper Review] Semantic Image Segmentation via Deep Parsing Network

Ziwei Liu, Xiaoxiao Li+3 more|arXiv (Cornell University)|Sep 9, 2015

Advanced Neural Network Applications45 references180 citations

TL;DR

This paper proposes Deep Parsing Network (DPN), a novel end-to-end convolutional neural network that unifies unary and pairwise terms in Markov Random Fields (MRFs) for semantic image segmentation. By approximating mean field inference in a single forward pass using learned convolutional layers, DPN achieves state-of-the-art 77.5% mIoU on PASCAL VOC 2012 without iterative MRF inference during backpropagation.

ABSTRACT

This paper addresses semantic image segmentation by incorporating rich information into Markov Random Field (MRF), including high-order relations and mixture of label contexts. Unlike previous works that optimized MRFs using iterative algorithm, we solve MRF by proposing a Convolutional Neural Network (CNN), namely Deep Parsing Network (DPN), which enables deterministic end-to-end computation in a single forward pass. Specifically, DPN extends a contemporary CNN architecture to model unary terms and additional layers are carefully devised to approximate the mean field algorithm (MF) for pairwise terms. It has several appealing properties. First, different from the recent works that combined CNN and MRF, where many iterations of MF were required for each training image during back-propagation, DPN is able to achieve high performance by approximating one iteration of MF. Second, DPN represents various types of pairwise terms, making many existing works as its special cases. Third, DPN makes MF easier to be parallelized and speeded up in Graphical Processing Unit (GPU). DPN is thoroughly evaluated on the PASCAL VOC 2012 dataset, where a single DPN model yields a new state-of-the-art segmentation accuracy.

Motivation & Objective

• Address the inefficiency of iterative MRF inference in deep learning-based semantic segmentation.
• Integrate high-order relations and label context mixtures into MRF pairwise potentials using a differentiable, end-to-end framework.
• Eliminate the need for recurrent or iterative MRF inference during backpropagation by approximating mean field with a single pass.
• Enable efficient, parallelizable, and GPU-accelerated inference by modeling MRFs as convolutional operations.
• Achieve state-of-the-art performance on PASCAL VOC 2012 while maintaining computational efficiency.

Proposed method

• Extend the VGG-16 architecture to model unary terms using features from pre-trained ImageNet weights.
• Introduce additional convolutional and pooling layers to approximate mean field inference for pairwise terms in MRFs.
• Model complex pairwise potentials, including high-order relations and label context mixtures, via learnable receptive fields.
• Formulate MRF inference as a deterministic, differentiable, single-pass computation, avoiding iterative optimization during backpropagation.
• Use low-rank approximation and GPU-friendly operations to accelerate inference and enable parallelization.
• Jointly train the entire network via backpropagation, end-to-end, with both unary and pairwise components optimized simultaneously.

Experimental results

Research questions

• RQ1 Can mean field inference in MRFs be approximated effectively in a single forward pass within a deep learning framework?
• RQ2 How does incorporating high-order relations and label context mixtures into pairwise potentials affect segmentation accuracy?
• RQ3 Can a unified, differentiable network architecture replace iterative MRF inference while maintaining or improving performance?
• RQ4 What is the impact of joint learning of unary and pairwise terms on segmentation accuracy and computational efficiency?
• RQ5 How does DPN generalize across object categories with varying sizes, shapes, and boundary complexities?

Key findings

• DPN achieves a new state-of-the-art mIoU of 77.5% on the PASCAL VOC 2012 test set using a single model.
• The model reduces runtime by at least 10× compared to prior methods requiring 10 iterations of mean field or RNN-based inference.
• Joint training significantly improves performance on most object categories, especially those with complex boundaries or contextual dependencies.
• The method achieves high tag accuracy (96.4%) through implicit modeling of image-level tags via pixel-wise supervision.
• Per-class analysis reveals that small objects (e.g., bird, cat, cow) are often discarded during joint tuning to improve overall smoothness and boundary accuracy.
• Object localization and boundary awareness are significantly enhanced in later training stages, particularly for objects with irregular shapes like 'bike'.