[Paper Review] ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-time

Rudra P. K. Poudel, Ujwal Bonde+2 more|arXiv (Cornell University)|May 11, 2018

Advanced Neural Network ApplicationsComputer Science18 references189 citations

TL;DR

ContextNet combines a deep low-resolution context branch with a shallow high-resolution detail branch to enable real-time semantic segmentation with low memory; it achieves 66.1% mIoU at 18.3 fps on Cityscapes full-res images.

ABSTRACT

Modern deep learning architectures produce highly accurate results on many challenging semantic segmentation datasets. State-of-the-art methods are, however, not directly transferable to real-time applications or embedded devices, since naive adaptation of such systems to reduce computational cost (speed, memory and energy) causes a significant drop in accuracy. We propose ContextNet, a new deep neural network architecture which builds on factorized convolution, network compression and pyramid representation to produce competitive semantic segmentation in real-time with low memory requirement. ContextNet combines a deep network branch at low resolution that captures global context information efficiently with a shallow branch that focuses on high-resolution segmentation details. We analyse our network in a thorough ablation study and present results on the Cityscapes dataset, achieving 66.1% accuracy at 18.3 frames per second at full (1024x2048) resolution (41.9 fps with pipelined computations for streamed data).

Motivation & Objective

• Motivate real-time semantic segmentation with low memory footprints for autonomous driving and embedded devices.
• Propose ContextNet architecture that fuses global context from a down-sampled branch with high-resolution local details.
• Evaluate the approach on the Cityscapes dataset with detailed ablations.
• Demonstrate that depth-wise separable convolutions and pruning enable efficient, accurate performance.

Proposed method

• Two-branch architecture with a low-resolution deep branch for global context and a high-resolution shallow branch for detail refinement.
• Use depth-wise separable convolutions and bottleneck residual blocks to reduce parameters and compute.
• Fuse branches via a fusion unit that adds features and applies a 1x1 convolution for final predictions.
• Apply an auxiliary loss on the low-resolution branch to encourage meaningful global-context features.
• Train with standard data augmentation and RMSprop optimization; use batch normalization and ReLU6 for robustness in low-precision settings.
• Prune the network post-training to explore smaller, faster variants (lottery-ticket style pruning).

Experimental results

Research questions

• RQ1 Can a two-branch network capturing global context at low resolution and local detail at full resolution achieve real-time semantic segmentation without substantial accuracy loss?
• RQ2 How do depth-wise convolutions and bottleneck blocks impact accuracy, speed, and memory on Cityscapes-scale data?
• RQ3 What is the impact of network pruning on mIoU and runtime for ContextNet on embedded/real-time settings?

Key findings

• ContextNet achieves 66.1% mIoU on Cityscapes test set after pruning.
• Without pruning, ContextNet achieves 64.2% mIoU with 18.3 fps on 1024×2048 images on a single CPU thread (Titan X measurement).
• The two-branch design with a deep low-resolution context branch and a shallow full-resolution detail branch balances accuracy and real-time performance.
• Pruning improves mIoU from 64.2% to 66.1% on Cityscapes test set.
• ContextNet runs at 18.3 fps at full resolution and can reach 41.9 fps with pipelined data in optimized setups.
• Compared to several real-time rivals, ContextNet offers competitive accuracy with lower memory footprint (0.85M parameters in the base variant).