[Paper Review] PP-LiteSeg: A Superior Real-Time Semantic Segmentation Model

Juncai Peng, Yi Liu+14 more|arXiv (Cornell University)|Apr 6, 2022

Advanced Neural Network Applications135 citations

TL;DR

PP-LiteSeg introduces three modules—Flexible and Lightweight Decoder (FLD), Unified Attention Fusion Module (UAFM), and Simple Pyramid Pooling Module (SPPM)—to achieve state-of-the-art accuracy-speed trade-offs for real-time semantic segmentation on Cityscapes and CamVid. It reports 72.0% mIoU at 273.6 FPS on Cityscapes test and 75.0% mIoU at 154.8 FPS on CamVid test.

ABSTRACT

Real-world applications have high demands for semantic segmentation methods. Although semantic segmentation has made remarkable leap-forwards with deep learning, the performance of real-time methods is not satisfactory. In this work, we propose PP-LiteSeg, a novel lightweight model for the real-time semantic segmentation task. Specifically, we present a Flexible and Lightweight Decoder (FLD) to reduce computation overhead of previous decoder. To strengthen feature representations, we propose a Unified Attention Fusion Module (UAFM), which takes advantage of spatial and channel attention to produce a weight and then fuses the input features with the weight. Moreover, a Simple Pyramid Pooling Module (SPPM) is proposed to aggregate global context with low computation cost. Extensive evaluations demonstrate that PP-LiteSeg achieves a superior trade-off between accuracy and speed compared to other methods. On the Cityscapes test set, PP-LiteSeg achieves 72.0% mIoU/273.6 FPS and 77.5% mIoU/102.6 FPS on NVIDIA GTX 1080Ti. Source code and models are available at PaddleSeg: https://github.com/PaddlePaddle/PaddleSeg.

Motivation & Objective

• Address the need for high-accuracy, real-time semantic segmentation suitable for real-world applications.
• Reduce decoder computation while balancing encoder-decoder efficiency.
• Strengthen feature representations through attention-guided fusion.
• Efficiently aggregate global context with low computational cost.
• Demonstrate state-of-the-art trade-offs on Cityscapes and CamVid datasets.

Proposed method

• Introduce Flexible and Lightweight Decoder (FLD) to gradually reduce feature channels from high-level to low-level features.
• Propose Unified Attention Fusion Module (UAFM) that uses spatial and channel attention to produce a fusion weight and fuse features via mul-add operations.
• Design Simple Pyramid Pooling Module (SPPM) that aggregates global context with reduced intermediate/output channels and uses addition instead of concatenation.
• Adopt SPPM to model long-range dependencies with low overhead.
• Build PP-LiteSeg with an encoder (STDC-based), SPPM, and FLD for aggregation and decoding; train with cross-entropy and Online Hard Example Mining.

Experimental results

Research questions

• RQ1 Can a lightweight decoder design balance encoder and decoder computation while maintaining high segmentation accuracy?
• RQ2 Does a unified attention-based fusion module improve feature representation at multi-level fusion efficiently?
• RQ3 Can a low-cost global context module (SPPM) maintain or improve accuracy without harming real-time performance?
• RQ4 What is the performance trade-off of PP-LiteSeg on standard real-time segmentation benchmarks (Cityscapes, CamVid) compared to prior methods?

Key findings

• PP-LiteSeg achieves a strong accuracy-speed trade-off on Cityscapes, reporting 72.0% mIoU with 273.6 FPS on test set for PP-LiteSeg-T1 and 77.5% mIoU with 102.6 FPS for PP-LiteSeg-B2 on test set.
• On Cityscapes, PP-LiteSeg-B2 reaches 78.2% mIoU at 102.6 FPS at 768×1536 resolution, compared to other real-time methods.
• On CamVid, PP-LiteSeg-T achieves 222.3 FPS, and PP-LiteSeg-B achieves 75.0% mIoU with 154.8 FPS.
• Ablation studies show FLD improves mIoU by 0.17% over a baseline, and combining FLD with SPPM and UAFM yields incremental gains, culminating in 78.21 mIoU at 102.6 FPS for PP-LiteSeg-B2.
• The proposed modules collectively deliver a state-of-the-art balance between accuracy and speed across Cityscapes and CamVid.