[Paper Review] BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation

Changqian Yu, Jingbo Wang+4 more|arXiv (Cornell University)|Aug 2, 2018

Advanced Neural Network ApplicationsComputer Science54 references126 citations

TL;DR

BiSeNet introduces a dual-path architecture (Spatial Path and Context Path) with a Feature Fusion Module and Attention Refinement Module to achieve real-time semantic segmentation with high accuracy. It reports 68.4% Mean IOU on Cityscapes test at 105 FPS on a Titan XP.

ABSTRACT

Semantic segmentation requires both rich spatial information and sizeable receptive field. However, modern approaches usually compromise spatial resolution to achieve real-time inference speed, which leads to poor performance. In this paper, we address this dilemma with a novel Bilateral Segmentation Network (BiSeNet). We first design a Spatial Path with a small stride to preserve the spatial information and generate high-resolution features. Meanwhile, a Context Path with a fast downsampling strategy is employed to obtain sufficient receptive field. On top of the two paths, we introduce a new Feature Fusion Module to combine features efficiently. The proposed architecture makes a right balance between the speed and segmentation performance on Cityscapes, CamVid, and COCO-Stuff datasets. Specifically, for a 2048x1024 input, we achieve 68.4% Mean IOU on the Cityscapes test dataset with speed of 105 FPS on one NVIDIA Titan XP card, which is significantly faster than the existing methods with comparable performance.

Motivation & Objective

• Motivate real-time semantic segmentation without sacrificing spatial detail or receptive field.
• Propose a Bilateral Segmentation Network (BiSeNet) comprising a Spatial Path and a Context Path to preserve spatial details and enlarge receptive field.
• Develop modules (Feature Fusion Module and Attention Refinement Module) to fuse and refine features efficiently.
• Evaluate BiSeNet on Cityscapes, CamVid, and COCO-Stuff to demonstrate speed-accuracy balance.
• Provide ablation studies to understand each component's impact on performance.

Proposed method

• Introduce Spatial Path (three 2-stride convolutions) to preserve high-resolution spatial information (1/8 scale).
• Introduce Context Path using a lightweight backbone (Xception39) with a U-shaped fusion of last two stages to provide a large receptive field.
• Add a Global Average Pooling branch on the tail of the Context Path to expand the receptive field with global context.
• Propose a Feature Fusion Module (FFM) that concatenates SP and CP features, applies BatchNorm, global pooling, and a SENet-like gating to reweight features.
• Propose an Attention Refinement Module (ARM) that uses global context to compute an attention vector to refine CP features without upsampling.
• Train with a joint loss: principal loss on the final output plus auxiliary losses on CP stages with a balancing parameter alpha = 1.
• Use an auxiliary deep supervision scheme for CP outputs to guide training.

Experimental results

Research questions

• RQ1 Can a dual-path architecture (preserving spatial details while providing a large receptive field) achieve real-time semantic segmentation without severe accuracy loss?
• RQ2 Do dedicated fusion and refinement modules (FFM and ARM) improve the speed-accuracy trade-off compared to standard single-path or U-shaped designs?
• RQ3 What is the contribution of global average pooling and auxiliary losses to the final performance of BiSeNet on standard benchmarks?

Key findings

• BiSeNet achieves strong real-time performance on Cityscapes, with 68.4% Mean IOU on the test set at high FPS (reported as 105 FPS on Titan XP for a 2048x1024 input).
• The Spatial Path alone improves mean IOU by approximately 1.4 percentage points over a CP-only baseline.
• The combination CP+SP with FFM and GP (global pooling) yields 68.42% and with ARM refinement reaches 71.40% in ablations on Cityscapes validation, illustrating progressive gains from module additions.
• Embedding global average pooling at the CP tail provides noticeable gains (67.42% to 68.42%).
• BiSeNet demonstrates competitive results on CamVid and COCO-Stuff, highlighting the method’s generalization across datasets.
• The approach maintains significantly higher speed than many prior methods with comparable accuracy on Cityscapes.