[Paper Review] Label Refinement Network for Coarse-to-Fine Semantic Segmentation

Amirul Islam, Shujon Naha+3 more|arXiv (Cornell University)|Mar 1, 2017

Advanced Neural Network Applications19 references51 citations

TL;DR

LRN introduces a coarse-to-fine CNN architecture that produces multi-resolution label maps with deep supervision at multiple decoder stages to improve pixel-wise semantic segmentation.

ABSTRACT

We consider the problem of semantic image segmentation using deep convolutional neural networks. We propose a novel network architecture called the label refinement network that predicts segmentation labels in a coarse-to-fine fashion at several resolutions. The segmentation labels at a coarse resolution are used together with convolutional features to obtain finer resolution segmentation labels. We define loss functions at several stages in the network to provide supervisions at different stages. Our experimental results on several standard datasets demonstrate that the proposed model provides an effective way of producing pixel-wise dense image labeling.

Motivation & Objective

• Introduce a coarse-to-fine strategy for semantic segmentation to preserve spatial precision while leveraging high-level encoder features.
• Develop an end-to-end CNN that predicts segmentation maps at multiple resolutions with losses at each stage.
• Incorporate refinement modules that fuse coarse label predictions with encoder features via skip connections to improve detail.
• Demonstrate effectiveness across standard datasets and show benefits over single-shot segmentation baselines.

Proposed method

• Encoder-decoder architecture based on VGG16-like encoder.
• Decoder includes refinement modules that combine upsampled segmentation maps with corresponding encoder features through skip connections.
• Multiple loss functions are applied at six decoder stages to supervise learning at different resolutions.
• Refinement module uses 3x3 convolutions, batch normalization, ReLU, concatenation of skip features, and bilinear upsampling.
• Final training optimizes the sum of all stage-wise losses: sum_k=1^6 l_k.

Experimental results

Research questions

• RQ1 Can a coarse-to-fine labeling strategy improve pixel-wise segmentation accuracy compared to single-shot full-resolution predictions?
• RQ2 Does deep supervision at multiple decoder stages and skip-connected refinement improve boundary sharpness and small object detail?
• RQ3 How does LRN perform relative to strong baselines (e.g., SegNet, FCN-8s) on standard semantic segmentation benchmarks?

Key findings

• LRN outperforms SegNet and SegNet+Deep Supervision on PASCAL VOC 2012, CamVid, and SUN RGB-D benchmarks.
• Progressive refinement with skip connections preserves fine details and improves small/thin objects (e.g., poles, signs) compared to baselines.
• Stage-wise supervision and coarse-to-fine predictions yield consistent gains across datasets without separate training for each stage.
• Ablation analysis shows that later-stage label maps (s_6) provide the best final performance, with progressively better mean IoU from early to late stages.
• Qualitative results illustrate improved boundary accuracy and finer object delineation over single-shot methods.