[Paper Review] Rethinking BiSeNet For Real-time Semantic Segmentation

Mingyuan Fan, Shenqi Lai+5 more|arXiv (Cornell University)|Apr 27, 2021

Advanced Neural Network ApplicationsComputer Science32 references39 citations

TL;DR

The paper introduces STDC-Seg with a Short-Term Dense Concatenate backbone and a Detail Guidance decoder to enable fast, accurate real-time semantic segmentation without extra inference cost.

ABSTRACT

BiSeNet has been proved to be a popular two-stream network for real-time segmentation. However, its principle of adding an extra path to encode spatial information is time-consuming, and the backbones borrowed from pretrained tasks, e.g., image classification, may be inefficient for image segmentation due to the deficiency of task-specific design. To handle these problems, we propose a novel and efficient structure named Short-Term Dense Concatenate network (STDC network) by removing structure redundancy. Specifically, we gradually reduce the dimension of feature maps and use the aggregation of them for image representation, which forms the basic module of STDC network. In the decoder, we propose a Detail Aggregation module by integrating the learning of spatial information into low-level layers in single-stream manner. Finally, the low-level features and deep features are fused to predict the final segmentation results. Extensive experiments on Cityscapes and CamVid dataset demonstrate the effectiveness of our method by achieving promising trade-off between segmentation accuracy and inference speed. On Cityscapes, we achieve 71.9% mIoU on the test set with a speed of 250.4 FPS on NVIDIA GTX 1080Ti, which is 45.2% faster than the latest methods, and achieve 76.8% mIoU with 97.0 FPS while inferring on higher resolution images.

Motivation & Objective

• Design an efficient backbone (STDC module) that provides scalable receptive fields with fewer parameters and computations.
• Eliminate the need for an extra spatial path by integrating spatial detail learning into low-level features via Detail Guidance.
• Fuse high-level semantic features with guided low-level details to improve boundary preservation.
• Achieve a strong speed-accuracy trade-off on real-time segmentation benchmarks (Cityscapes, CamVid) using an end-to-end single-stream architecture.

Proposed method

• Introduce STDC module that concatenates multi-scale feature maps from several blocks with progressively smaller kernels to obtain scalable receptive fields and reduced FLOPs.
• Down-sample and fuse STDC block outputs via concatenation after down-sampling intermediate maps to a common resolution.
• In the decoder, apply Detail Guidance through a Detail Aggregation module that generates binary detail ground-truth from a Laplacian-based process and a Detail Head to guide low-level features without adding inference cost.
• Train with a segmentation loss plus a detail loss (L_detail = L_Dice + L_BCE) to steer low-level features toward spatial details; inference uses only the segmentation path.
• Use a BiSeNet-inspired context path for multi-scale context and fuse with the decoder through a Feature Fusion module.

Figure 1: Speed-Accuracy performance comparison on the Cityscapes test set. Our methods are presented in red dots while other methods are presented in blue dots. Our approaches achieve state-of-the-art speed-accuracy trade-off.

Experimental results

Research questions

• RQ1 Can the proposed STDC backbone achieve competitive segmentation accuracy with significantly higher speed compared to lightweight backbones for real-time segmentation?
• RQ2 Does the Detail Guidance module improve boundary and small object delineation without increasing inference time?
• RQ3 How does STDC-Seg compare with state-of-the-art real-time segmentation methods on Cityscapes and CamVid in terms of mIoU and FPS?
• RQ4 What is the impact of the number of STDC blocks on accuracy and speed?
• RQ5 Is the single-stream decoder with Detail Guidance sufficient to replace BiSeNet-style two-path architectures?

Key findings

• STDC2-Seg50 achieves 73.4% mIoU at 188.6 FPS on Cityscapes with 512x1024 input.
• STDC1-Seg50 achieves 71.9% mIoU at 250.4 FPS on Cityscapes with 512x1024 input.
• STDC2-Seg75 achieves 76.8% mIoU at 97.0 FPS on Cityscapes with 768x1536 input.
• On Cityscapes validation, STDC2-Seg75 achieves 77.0% mIoU with 97.0 FPS (768x1536 inputs).
• STDC1-Seg achieves 73.0% mIoU at 197.6 FPS on CamVid (720x960).
• Detail Guidance improves boundary and small-object delineation without increasing inference cost, outperforming Spatial Path-based configurations in accuracy per compute.

Figure 2: Illustration of architectures of BiSeNet [ 28 ] and our proposed approach. (a) presents Bilateral Segmentation Network (BiSeNet [ 28 ] ), which use an extra Spatial Path to encode spatial information. (b) demonstrates our proposed method, which use a Detail Guidance module to encode spatia