[Paper Review] DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Liang-Chieh Chen, George Papandreou+3 more|arXiv (Cornell University)|Jun 2, 2016

Advanced Neural Network Applications85 references710 citations

TL;DR

Presents DeepLab, a semantic segmentation system using atrous convolution for dense features, ASPP for multi-scale context, and a fully connected CRF for boundary refinement, achieving state-of-the-art results on PASCAL VOC 2012 and other datasets.

ABSTRACT

In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First, we highlight convolution with upsampled filters, or 'atrous convolution', as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second, we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third, we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed "DeepLab" system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7% mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.

Motivation & Objective

• Motivate and address three challenges in applying DCNNs to semantic segmentation: (i) reduced feature resolution, (ii) multi-scale object sizes, and (iii) localization accuracy.
• Propose atrous convolution to control feature resolution and enlarge receptive field without extra parameters.
• Introduce atrous Spatial Pyramid Pooling (ASPP) to capture multi-scale context efficiently.
• Improve boundary localization by integrating a fully connected CRF on top of DCNN outputs.

Proposed method

• Use a fully convolutional network with atrous convolution to compute dense feature maps at higher resolution than standard DCNNs.
• Replace standard multilayer downsampling with atrous convolution to enlarge the field of view without increasing parameters.
• Implement ASPP as parallel atrous convolutions with different rates to capture multi-scale context.
• Upsample the final DCNN score maps by bilinear interpolation to original image size, then refine with a fully connected CRF for sharp boundaries.
• Fine-tune Imagenet-pretrained networks (VGG-16 or ResNet-101) for semantic segmentation, and train CRF parameters separately on a validation set.
• Provide publicly available code and models extending the Caffe framework.

Experimental results

Research questions

• RQ1 Can atrous convolution enable high-resolution dense predictions without extra parameters or computation?
• RQ2 Does atrous spatial pyramid pooling improve segmentation across object scales more efficiently than multi-scale image processing?
• RQ3 Does combining DCNN outputs with a fully connected CRF improve boundary localization and overall segmentation accuracy?
• RQ4 What is the impact of deeper networks (VGG-16 vs. ResNet-101) on semantic segmentation performance when using atrous convolution and ASPP?

Key findings

• DeepLab with atrous convolution achieves high-resolution feature maps (8x sampling) with bilinear upsampling to original image size.
• ASPP improves multi-scale object segmentation by probing features at multiple sampling rates.
• Fully connected CRF refines boundaries and improves localization, yielding ~3–5% absolute gains in mean IOU across variants.
• On PASCAL VOC 2012, DeepLab-CRF-LargeFOV reaches 70.3% mean IOU on the official test set, with state-of-the-art results reported on VOC 2012, PASCAL-Context, PASCAL-Person-Part, and Cityscapes.
• Using a larger field of view (Small kernel with high rate) plus CRF provides best accuracy-speed trade-offs (e.g., DeepLab-LargeFOV).
• DeepLab variants with AS PP and CRF outperform conference-version results and show gains with deeper networks and multi-scale processing.