[Paper Review] Context Encoding for Semantic Segmentation

Hang Zhang, Kristin Dana+5 more|arXiv (Cornell University)|Mar 23, 2018

Advanced Neural Network ApplicationsComputer Science51 references135 citations

TL;DR

Introduces a Context Encoding Module (EncNet) that leverages global scene context via an Encoding Layer and a Semantic Encoding Loss to improve semantic segmentation with minimal extra computation, achieving state-of-the-art results on PASCAL VOC 2012, PASCAL-Context, and ADE20K.

ABSTRACT

Recent work has made significant progress in improving spatial resolution for pixelwise labeling with Fully Convolutional Network (FCN) framework by employing Dilated/Atrous convolution, utilizing multi-scale features and refining boundaries. In this paper, we explore the impact of global contextual information in semantic segmentation by introducing the Context Encoding Module, which captures the semantic context of scenes and selectively highlights class-dependent featuremaps. The proposed Context Encoding Module significantly improves semantic segmentation results with only marginal extra computation cost over FCN. Our approach has achieved new state-of-the-art results 51.7% mIoU on PASCAL-Context, 85.9% mIoU on PASCAL VOC 2012. Our single model achieves a final score of 0.5567 on ADE20K test set, which surpass the winning entry of COCO-Place Challenge in 2017. In addition, we also explore how the Context Encoding Module can improve the feature representation of relatively shallow networks for the image classification on CIFAR-10 dataset. Our 14 layer network has achieved an error rate of 3.45%, which is comparable with state-of-the-art approaches with over 10 times more layers. The source code for the complete system are publicly available.

Motivation & Objective

• Motivate the use of global scene context to reduce the search space of possible object categories during segmentation.
• Develop a lightweight module that encodes global feature statistics and selectively scales class-dependent feature maps.
• Regularize training with a Semantic Encoding Loss to encourage recognition of present categories in the scene.
• Integrate the Context Encoding Module into a Dilated FCN backbone (EncNet) and evaluate on standard benchmarks.
• Demonstrate additional benefits of context encoding for shallow networks on image classification (CIFAR-10).

Proposed method

• Extend Encoding Layer to capture global context statistics from dense convolutional features.
• Predict channel-wise scaling factors for feature maps from the Encoding Layer output and apply them via elementwise multiplication.
• Introduce Semantic Encoding Loss (SE-loss) that predicts presence of object categories in the scene to regularize training.
• Build EncNet by inserting the Context Encoding Module on top of pre-trained ResNet with dilated convolutions, and optionally apply SE-loss at multiple stages.
• Use synchronized cross-GPU Batch Normalization to stabilize training with larger effective batch sizes.
• Evaluate under standard metrics (pixAcc, mIoU) on PASCAL-Context, PASCAL VOC 2012, and ADE20K; also test CIFAR-10 classification performance.

Experimental results

Research questions

• RQ1 Does explicit global context modeling improve per-pixel semantic segmentation beyond receptive field enlargement techniques?
• RQ2 Can a lightweight Context Encoding Module improve segmentation without substantial computational overhead?
• RQ3 How does SE-loss influence learning of scene-level semantics and small object segmentation?
• RQ4 Is EncNet competitive with or superior to state-of-the-art methods on PASCAL-Context, VOC 2012, and ADE20K without COCO pre-training?
• RQ5 Can context encoding benefits extend to shallower networks for image classification (CIFAR-10)?

Key findings

• EncNet with the Context Encoding Module yields significant gains over the FCN baseline (e.g., from 41.0% mIoU to 47.6% mIoU on a ResNet-50 baseline).
• With ResNet-101, EncNet achieves 51.7% mIoU on PASCAL-Context and 85.9% mIoU on PASCAL VOC 2012 (with COCO pre-training in the VOC case).
• EncNet-101 single model achieves 0.5567 on ADE20K test set, surpassing the COCO-Place Challenge 2017 winner.
• SE-loss weight of 0.2 and 32 codewords in the Encoding Layer provide best ablation performance, with marginal additional computation.
• On CIFAR-10, a 14-layer EncNet achieves 3.96% error, competitive with deeper models, demonstrating broader utility of context encoding.
• EncNet offers state-of-the-art results on major segmentation benchmarks while preserving efficiency and compatibility with existing FCN-based frameworks.