[Paper Review] Fully Convolutional Networks for Semantic Segmentation

J. D. Long, Evan Shelhamer+1 more|arXiv (Cornell University)|Nov 14, 2014

Advanced Neural Network Applications2,812 citations

TL;DR

The paper converts classification nets into fully convolutional networks (FCNs) to perform end-to-end, pixel-wise semantic segmentation, achieving state-of-the-art results on multiple datasets with efficient, end-to-end training and a skip-based architecture that fuses multi-scale information.

ABSTRACT

Convolutional networks are powerful visual models that yield hierarchies of features. We show that convolutional networks by themselves, trained end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic segmentation. Our key insight is to build "fully convolutional" networks that take input of arbitrary size and produce correspondingly-sized output with efficient inference and learning. We define and detail the space of fully convolutional networks, explain their application to spatially dense prediction tasks, and draw connections to prior models. We adapt contemporary classification networks (AlexNet, the VGG net, and GoogLeNet) into fully convolutional networks and transfer their learned representations by fine-tuning to the segmentation task. We then define a novel architecture that combines semantic information from a deep, coarse layer with appearance information from a shallow, fine layer to produce accurate and detailed segmentations. Our fully convolutional network achieves state-of-the-art segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012), NYUDv2, and SIFT Flow, while inference takes one third of a second for a typical image.

Motivation & Objective

• Motivate and demonstrate that fully convolutional networks trained end-to-end can perform pixelwise semantic segmentation without extra post-processing or proposals.
• Adapt and repurpose existing classification nets (AlexNet, VGG, GoogLeNet) into FCNs suitable for dense prediction through end-to-end fine-tuning.
• Develop a skip architecture (FCN-32s, FCN-16s, FCN-8s) that combines coarse semantic information with fine appearance details to improve spatial precision.
• Evaluate the approach on standard datasets (PASCAL VOC 2011/2012, NYUDv2, SIFT Flow) and compare against prior state-of-the-art methods.
• Analyze training and inference efficiency of full-image FCN training versus patch-based methods and demonstrate end-to-end learning of upsampling filters.

Proposed method

• Transform traditional classification nets into fully convolutional networks by converting fully connected layers to convolutions and appending a 1x1 convolution to produce class scores per location.
• Add in-network upsampling (deconvolution) layers to recover dense pixel-wise predictions from coarse outputs and learn these upsampling filters jointly via backpropagation.
• Introduce a skip architecture that fuses predictions from multiple layers (pool4/ pool3 and conv7) to enhance spatial detail while preserving high-level semantics (FCN-32s, FCN-16s, FCN-8s).
• Fine-tune the adapted networks on segmentation data, using per-pixel multinomial logistic loss and mean intersection-over-union (IU) as the evaluation metric.
• Compare single-stream and skip-connected FCNs and report inference-time improvements and accuracy gains on multiple datasets.

Experimental results

Research questions

• RQ1 Can end-to-end trained FCNs outperform state-of-the-art semantic segmentation approaches without external post-processing or proposals?
• RQ2 Does adapting classification nets into FCNs with in-network upsampling enable accurate dense predictions for segmentation tasks?
• RQ3 Does combining coarse deep features with fine shallow features via a skip architecture improve segmentation detail and accuracy?
• RQ4 What is the impact of end-to-end fine-tuning and multi-scale fusion on standard segmentation benchmarks (PASCAL VOC, NYUDv2, SIFT Flow)?

Key findings

• FCN-8s achieves a mean IU of 62.7 on PASCAL VOC 2011 test and 62.2 on VOC 2012 test, outperforming the prior state-of-the-art SDS by about 20% relative.
• FCN-16s and FCN-8s improve over FCN-32s, with mean IU rising from 59.4 to 62.4 and 62.7 respectively on PASCAL VOC validation, demonstrating gains from skip connections.
• On NYUDv2, RGB-HHA and RGB-HHA fused models reach mean IU of 32.8 and 34.0 respectively, surpassing prior methods; RGB-HHA with FCN-16s achieves 34.0 mean IU.
• On SIFT Flow, FCN-16s (mean IU 39.5) and FCN-8s (mean IU - as reported in table) demonstrate competitive performance with strong semantic and geometric labeling results.
• End-to-end training with in-network upsampling yields fast inference (as low as ~175 ms for a 500x500 input) and removes the need for post-processing steps such as superpixels or CRFs.
• Combining coarse semantic information with fine appearance information via skip connections yields refined segmentations with improved boundary detail and spatial precision.