[Paper Review] Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Liang-Chieh Chen, George Papandreou+3 more|arXiv (Cornell University)|Dec 22, 2014

Advanced Neural Network ApplicationsComputer Science64 references3,623 citations

TL;DR

The paper combines deep CNNs with fully connected CRFs to perform pixel-level semantic segmentation, achieving state-of-the-art results on PASCAL VOC 2012 by addressing localization with a dense CRF and improving efficiency via the atrous (hole) algorithm.

ABSTRACT

Deep Convolutional Neural Networks (DCNNs) have recently shown state of the art performance in high level vision tasks, such as image classification and object detection. This work brings together methods from DCNNs and probabilistic graphical models for addressing the task of pixel-level classification (also called "semantic image segmentation"). We show that responses at the final layer of DCNNs are not sufficiently localized for accurate object segmentation. This is due to the very invariance properties that make DCNNs good for high level tasks. We overcome this poor localization property of deep networks by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF). Qualitatively, our "DeepLab" system is able to localize segment boundaries at a level of accuracy which is beyond previous methods. Quantitatively, our method sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 71.6% IOU accuracy in the test set. We show how these results can be obtained efficiently: Careful network re-purposing and a novel application of the 'hole' algorithm from the wavelet community allow dense computation of neural net responses at 8 frames per second on a modern GPU.

Motivation & Objective

• Motivate pixel-level semantic segmentation where DCNNs’ invariance hurts localization.
• Repurpose a pretrained CNN into a dense feature extractor with minimal loss of performance.
• Address localization by coupling DCNN unary terms with a fully connected CRF for boundary refinement.

Proposed method

• Convert a 16-layer CNN (VGG-16) to a fully convolutional network for dense, per-pixel scoring.
• Use the hole (atrous) algorithm to compute dense feature maps at 8-pixel stride, enabling high-resolution score maps.
• Fine-tune the CNN on PASCAL VOC 2012 for 21-class pixel labeling, using per-pixel cross-entropy over the dense output.
• Attach a fully connected CRF (Krähenbühl & Koltun 2011) with bilateral and spatial kernels to refine segmentation boundaries.
• Optionally incorporate multi-scale features from intermediate CNN layers to boost localization.
• Explore variations with different field-of-view settings (atrous kernel sizes and strides) to balance accuracy and speed.

Experimental results

Research questions

• RQ1 Can a fully-connected CRF improve localization when combined with a DCNN-based unary term for semantic segmentation?
• RQ2 What is the impact of multi-scale features and large field-of-view on segmentation performance?
• RQ3 How does the atrous convolution approach affect dense score computation and overall efficiency?
• RQ4 What trade-offs exist between accuracy, speed, and model size when varying receptive field and FOV?

Key findings

• Achieves state-of-the-art mean IOU on PASCAL VOC 2012 test set (71.6% with DeepLab-MSc-CRF-LargeFOV).
• Adding a fully connected CRF to the DeepLab system yields substantial performance gains (e.g., DeepLab-CRF improves over DeepLab by ~4% on validation).
• Multi-scale features provide additional localization benefits, further improving IoU when combined with CRF.
• The atrous (hole) algorithm enables dense 8-pixel stride CNN scoring at 8 fps on a modern GPU, while CRF inference runs in under 0.5 seconds on average for VOC images.
• Large Field-of-View variants achieve competitive or superior accuracy with faster runtimes compared to some alternatives, demonstrating effective trade-offs for real-time or near real-time applications.
• Best single model on the VOC 2012 test set (DeepLab-MSc-CRF-LargeFOV) reaches 71.6% mean IOU, surpassing FCN-8s and other contemporaries.