[Paper Review] Fast-SCNN: Fast Semantic Segmentation Network

Rudra P. K. Poudel, Stephan Liwicki+1 more|arXiv (Cornell University)|Feb 12, 2019

Advanced Neural Network Applications26 references364 citations

TL;DR

Fast-SCNN delivers above-real-time semantic segmentation for high-resolution images with a shared early feature extractor, achieving 68.0% mIoU at 123.5 fps on Cityscapes using 1.11M parameters, with minimal benefit from ImageNet pre-training.

ABSTRACT

The encoder-decoder framework is state-of-the-art for offline semantic image segmentation. Since the rise in autonomous systems, real-time computation is increasingly desirable. In this paper, we introduce fast segmentation convolutional neural network (Fast-SCNN), an above real-time semantic segmentation model on high resolution image data (1024x2048px) suited to efficient computation on embedded devices with low memory. Building on existing two-branch methods for fast segmentation, we introduce our `learning to downsample' module which computes low-level features for multiple resolution branches simultaneously. Our network combines spatial detail at high resolution with deep features extracted at lower resolution, yielding an accuracy of 68.0% mean intersection over union at 123.5 frames per second on Cityscapes. We also show that large scale pre-training is unnecessary. We thoroughly validate our metric in experiments with ImageNet pre-training and the coarse labeled data of Cityscapes. Finally, we show even faster computation with competitive results on subsampled inputs, without any network modifications.

Motivation & Objective

• Motivate real-time semantic segmentation for high-resolution imagery and embedded devices.
• Introduce a shared early feature extractor (learning to downsample) to combine detail and context efficiently.
• Design a low-capacity network (1.11M parameters) using depthwise separable convolutions and inverted residual blocks.
• Demonstrate that pre-training on ImageNet offers limited gains for this low-capacity model.

Proposed method

• Propose a fast segmentation network (Fast-SCNN) with a learning to downsample module that shares early convolutions across two resolution branches.
• Use a coarse global feature extractor to capture context at reduced resolution with residual bottleneck blocks.
• Incorporate a feature fusion module to combine high-resolution spatial details with low-resolution global context via simple addition.
• Employ depthwise separable convolutions and inverted residual blocks to reduce parameters and FLOPs.
• Include a classifier head with a small stack of depthwise separable convolutions and a possible softmax or argmax inference option.

Experimental results

Research questions

• RQ1 How can real-time semantic segmentation be achieved on high-resolution images with embedded devices without high memory requirements?
• RQ2 Does sharing early layer computations between resolution branches (learning to downsample) preserve accuracy while boosting speed?
• RQ3 What is the impact of network capacity and pre-training on Cityscapes performance for a lightweight model?

Key findings

• Fast-SCNN achieves 68.0% mIoU on Cityscapes at 123.5 fps on a Titan Xp (Pascal) with 1024x2048 input.
• The model uses about 1.11 million parameters, significantly fewer than many real-time and offline methods.
• A learning to downsample module and a single skip connection enable efficient multi-resolution feature sharing and boundary preservation.
• Pre-training on ImageNet or adding coarse Cityscapes data yields only marginal gains for this low-capacity network (≈0.5% mIoU).
• Lowering input resolution increases FPS (e.g., 1024x2048: 123.5 fps; 512x1024: 285.8 fps; 256x512: 485.4 fps) with corresponding mIoUs (68.0%, 62.8%, 51.9%).