[Paper Review] FDDWNet: A Lightweight Convolutional Neural Network for Real-time Sementic Segmentation

Jia Liu, Quan Zhou+4 more|arXiv (Cornell University)|Nov 2, 2019

Advanced Neural Network Applications26 references4 citations

TL;DR

FDDWNet is a lightweight, deep encoder-decoder CNN for real-time semantic segmentation that uses factorized dilated depth-wise separable convolutions (FDDWC) and multi-branch skip connections to achieve high accuracy with minimal parameters. It achieves 60 FPS inference speed and 71.5% mIoU on CityScapes with only 0.8M parameters, outperforming prior lightweight models in speed-accuracy trade-off.

ABSTRACT

This paper introduces a lightweight convolutional neural network, called FDDWNet, for real-time accurate semantic segmentation. In contrast to recent advances of lightweight networks that prefer to utilize shallow structure, FDDWNet makes an effort to design more deeper network architecture, while maintains faster inference speed and higher segmentation accuracy. Our network uses factorized dilated depth-wise separable convolutions (FDDWC) to learn feature representations from different scale receptive fields with fewer model parameters. Additionally, FDDWNet has multiple branches of skipped connections to gather context cues from intermediate convolution layers. The experiments show that FDDWNet only has 0.8M model size, while achieves 60 FPS running speed on a single RTX 2080Ti GPU with a 1024x512 input image. The comprehensive experiments demonstrate that our model achieves state-of-the-art results in terms of available speed and accuracy trade-off on CityScapes and CamVid datasets.

Motivation & Objective

• To address the trade-off between model efficiency and segmentation accuracy in real-time applications on edge devices.
• To design a deeper lightweight network architecture that maintains fast inference speed while improving feature representation.
• To reduce model size and computational cost without sacrificing segmentation performance.
• To explore the use of factorized dilated depth-wise separable convolutions for multi-scale receptive field learning with fewer parameters.
• To integrate multiple skip connections from intermediate layers to enhance context aggregation and model robustness.

Proposed method

• Proposes the Extremely Efficient Residual Module (EERM) using factorized dilated depth-wise separable convolutions (FDDWC) to reduce parameters and maintain high representational capacity.
• Employs FDDWC by decomposing standard convolutions into two 1D-factorized depth-wise convolutions with dilation, followed by 1×1 point-wise convolutions.
• Introduces multi-branch skip connections from intermediate layers to aggregate contextual features and improve gradient flow.
• Uses an end-to-end trainable architecture with identity mapping in EERM to prevent vanishing/exploding gradients.
• Applies a 'poly' learning rate policy with momentum and weight decay for stable training on CityScapes and CamVid datasets.
• Optimizes inference speed by using a small batch size (4) and efficient layer design, achieving 60 FPS on a single RTX 2080Ti.

Experimental results

Research questions

• RQ1 Can a deeper lightweight network architecture achieve better segmentation accuracy without increasing model size or inference latency?
• RQ2 How effective is factorized dilated depth-wise separable convolution (FDDWC) in learning multi-scale features with fewer parameters?
• RQ3 What is the impact of multi-branch skip connections from intermediate layers on context aggregation and performance in real-time segmentation?
• RQ4 How does FDDWNet compare to state-of-the-art lightweight models in terms of speed-accuracy trade-off on standard benchmarks?
• RQ5 Can a lightweight network with 0.8M parameters achieve state-of-the-art performance on CityScapes and CamVid without data augmentation or post-processing?

Key findings

• FDDWNet achieves 71.5% mIoU on the CityScapes validation set, outperforming most existing lightweight models.
• It runs at 60 FPS on a single RTX 2080Ti GPU with a 1024×512 input, demonstrating real-time inference capability.
• The model has only 0.8M parameters, making it 53× smaller than ICNet while being nearly 2× faster.
• FDDWNet achieves the best mIoU for 13 out of 19 categories on CityScapes, with notable improvements of 6.1% for 'Wall' and 2.7% for 'Pedestrian'.
• On CamVid, FDDWNet achieves 66.9% mIoU and runs at 79 FPS, outperforming ICNet in speed with only a 0.2% accuracy drop.
• Despite its deeper architecture, FDDWNet maintains comparable model size to DABNet but achieves nearly 2× deeper network depth, indicating superior parameter efficiency.