[Paper Review] Learning Deconvolution Network for Semantic Segmentation

Hyeonwoo Noh, Seunghoon Hong+1 more|arXiv (Cornell University)|May 17, 2015

Advanced Neural Network Applications25 references612 citations

TL;DR

This paper proposes a deconvolution network for semantic segmentation that learns to reconstruct dense, high-resolution segmentation masks by reversing the feature hierarchy of a pre-trained VGG-16 network. By applying the trained network to object proposals and combining results, it overcomes scale limitations of fully convolutional networks, achieving state-of-the-art performance (72.5% mIoU) on PASCAL VOC 2012 without external data, with further gains via ensemble with FCN-8s.

ABSTRACT

We propose a novel semantic segmentation algorithm by learning a deconvolution network. We learn the network on top of the convolutional layers adopted from VGG 16-layer net. The deconvolution network is composed of deconvolution and unpooling layers, which identify pixel-wise class labels and predict segmentation masks. We apply the trained network to each proposal in an input image, and construct the final semantic segmentation map by combining the results from all proposals in a simple manner. The proposed algorithm mitigates the limitations of the existing methods based on fully convolutional networks by integrating deep deconvolution network and proposal-wise prediction; our segmentation method typically identifies detailed structures and handles objects in multiple scales naturally. Our network demonstrates outstanding performance in PASCAL VOC 2012 dataset, and we achieve the best accuracy (72.5%) among the methods trained with no external data through ensemble with the fully convolutional network.

Motivation & Objective

• To address scale invariance and coarse feature representation issues in fully convolutional network (FCN)-based semantic segmentation.
• To improve fine-grained object detail recovery by learning a multi-layer deconvolution network with deconvolution and unpooling layers.
• To overcome the fixed receptive field limitation of FCNs by using instance-wise prediction on object proposals.
• To achieve state-of-the-art performance on PASCAL VOC 2012 using only the VOC 2012 dataset and model ensembling.

Proposed method

• A deep deconvolution network is trained on top of VGG-16 features, using deconvolution, unpooling, and ReLU layers to upsample feature maps progressively.
• The network is trained end-to-end using supervised learning with ground-truth segmentation masks as targets.
• Object proposals are extracted from the input image and each is fed into the deconvolution network to generate instance-wise segmentation masks.
• The final segmentation map is constructed by combining predictions from all proposals using a simple, non-differentiable fusion strategy.
• Model ensembling with FCN-8s is applied to exploit complementary strengths between the two architectures.
• ACRF is optionally applied to refine outputs, though it yields only marginal quantitative improvement.

Experimental results

Research questions

• RQ1 Can a learned deconvolution network reconstruct detailed, high-resolution segmentation masks more effectively than bilinear interpolation in FCNs?
• RQ2 Does instance-wise prediction on object proposals mitigate scale-related misclassification and fragmentation in semantic segmentation?
• RQ3 Can a deconvolution network trained solely on PASCAL VOC 2012 achieve state-of-the-art performance without external data?
• RQ4 How do the complementary characteristics of the proposed deconvolution network and FCN-8s improve performance when ensembled?

Key findings

• The proposed deconvolution network achieves 72.5% mean Intersection over Union (mIoU) on the PASCAL VOC 2012 validation set, the best result among methods trained without external data.
• Ensembling the deconvolution network with FCN-8s further improves performance, correcting inaccurate predictions from both models and yielding superior results.
• The method effectively recovers fine object structures and handles multi-scale objects better than FCN-based approaches, especially for small or fragmented objects.
• While CRF refinement reduces noise, it provides only marginal improvement in quantitative metrics, indicating the deconvolution network already produces high-quality outputs.
• The network demonstrates robustness to misaligned or background proposals, though such cases occasionally lead to noisy predictions.
• The architecture’s progressive deconvolution and unpooling operations enable coarse-to-fine reconstruction of object shapes, enhancing localization accuracy.