[Paper Review] ESNet: An Efficient Symmetric Network for Real-time Semantic Segmentation

Yu Wang, Quan Zhou+1 more|arXiv (Cornell University)|Jun 24, 2019

Advanced Neural Network Applications38 references4 citations

TL;DR

ESNet proposes a lightweight, symmetric encoder-decoder network for real-time semantic segmentation using a novel parallel factorized convolution unit (PFCU) that applies a transform-split-transform-merge strategy to reduce computational cost while preserving feature representation. With only 1.6M parameters, it achieves 62 FPS on a single GTX 1080Ti and sets a new state-of-the-art accuracy-efficiency trade-off on the CityScapes dataset, outperforming prior lightweight models in both speed and mIOU scores.

ABSTRACT

The recent years have witnessed great advances for semantic segmentation using deep convolutional neural networks (DCNNs). However, a large number of convolutional layers and feature channels lead to semantic segmentation as a computationally heavy task, which is disadvantage to the scenario with limited resources. In this paper, we design an efficient symmetric network, called (ESNet), to address this problem. The whole network has nearly symmetric architecture, which is mainly composed of a series of factorized convolution unit (FCU) and its parallel counterparts (PFCU). On one hand, the FCU adopts a widely-used 1D factorized convolution in residual layers. On the other hand, the parallel version employs a transform-split-transform-merge strategy in the designment of residual module, where the split branch adopts dilated convolutions with different rate to enlarge receptive field. Our model has nearly 1.6M parameters, and is able to be performed over 62 FPS on a single GTX 1080Ti GPU. The experiments demonstrate that our approach achieves state-of-the-art results in terms of speed and accuracy trade-off for real-time semantic segmentation on CityScapes dataset.

Motivation & Objective

• To address the challenge of deploying accurate semantic segmentation models on resource-constrained devices such as drones, robots, and smartphones.
• To reduce computational complexity and model size without sacrificing segmentation accuracy, especially for real-time inference.
• To design a symmetric, efficient architecture that maintains high feature representation capability through novel residual block design.
• To achieve a superior balance between speed, accuracy, and model compactness compared to existing lightweight networks.

Proposed method

• The network employs a nearly symmetric encoder-decoder architecture based on residual blocks, with down-sampling and up-sampling units for feature hierarchy.
• The core component is the parallel factorized convolution unit (PFCU), which uses a transform-split-transform-merge strategy to decompose 2D convolutions into 1D factorized convolutions across multiple parallel paths.
• The PFCU integrates identity shortcuts and multi-path 1D convolutions with different dilation rates to expand receptive fields while reducing FLOPs.
• Factorized convolution units (FCUs) use 1D factorized convolutions with adaptive kernel sizes to capture multi-scale features efficiently.
• The architecture is end-to-end trainable and can be directly applied to existing ResNet-based segmentation and classification networks.
• The design enables significant parameter reduction and faster inference, with minimal accuracy degradation.

Experimental results

Research questions

• RQ1 Can a symmetric, lightweight network architecture achieve high segmentation accuracy while maintaining real-time inference speed on edge devices?
• RQ2 How does the transform-split-transform-merge strategy in the PFCU improve feature representation efficiency compared to standard 2D convolutions?
• RQ3 What is the trade-off between model size, inference speed, and segmentation mIOU when using factorized and parallel convolutional designs?
• RQ4 How does ESNet compare to state-of-the-art lightweight networks like ENet, ESPNet, and CGNet in terms of accuracy and speed?

Key findings

• ESNet achieves 62 FPS inference speed on a single GTX 1080Ti GPU with only 1.6 million parameters, significantly outperforming SegNet in both speed and model size.
• On the CityScapes validation set, ESNet achieves 69.1% class mIOU and 69.1% category mIOU without data augmentation, matching ICNet’s performance with a 0.4% lower class mIOU but higher category mIOU.
• With 20K additional coarse annotations, ESNet achieves 70.7% class mIOU and 87.4% category mIOU, setting new state-of-the-art results in accuracy-efficiency trade-off.
• ESNet is nearly 4× faster and 18× smaller than SegNet, and despite having 5× more parameters than ENet, it achieves 12.4% and 7% higher mIOU scores in class and category mIOU, respectively.
• Visual results show that ESNet generalizes well across object scales, particularly improving detection of small objects like traffic signs and traffic lights.
• The PFCU design enables efficient inference with reduced FLOPs, and the symmetric architecture simplifies training and inference while maintaining high performance.