[Paper Review] BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation

Changqian Yu, Changxin Gao+4 more|arXiv (Cornell University)|Apr 5, 2020

Advanced Neural Network ApplicationsComputer Science66 references115 citations

TL;DR

BiSeNet V2 introduces a two-pathway architecture (Detail Branch for spatial details and Semantic Branch for semantics) with a Bilateral Guided Aggregation Layer and booster training to achieve high-accuracy real-time semantic segmentation, e.g., 72.6% mIoU at 156 FPS on Cityscapes test."

ABSTRACT

The low-level details and high-level semantics are both essential to the semantic segmentation task. However, to speed up the model inference, current approaches almost always sacrifice the low-level details, which leads to a considerable accuracy decrease. We propose to treat these spatial details and categorical semantics separately to achieve high accuracy and high efficiency for realtime semantic segmentation. To this end, we propose an efficient and effective architecture with a good trade-off between speed and accuracy, termed Bilateral Segmentation Network (BiSeNet V2). This architecture involves: (i) a Detail Branch, with wide channels and shallow layers to capture low-level details and generate high-resolution feature representation; (ii) a Semantic Branch, with narrow channels and deep layers to obtain high-level semantic context. The Semantic Branch is lightweight due to reducing the channel capacity and a fast-downsampling strategy. Furthermore, we design a Guided Aggregation Layer to enhance mutual connections and fuse both types of feature representation. Besides, a booster training strategy is designed to improve the segmentation performance without any extra inference cost. Extensive quantitative and qualitative evaluations demonstrate that the proposed architecture performs favourably against a few state-of-the-art real-time semantic segmentation approaches. Specifically, for a 2,048x1,024 input, we achieve 72.6% Mean IoU on the Cityscapes test set with a speed of 156 FPS on one NVIDIA GeForce GTX 1080 Ti card, which is significantly faster than existing methods, yet we achieve better segmentation accuracy.

Motivation & Objective

• Motivate real-time semantic segmentation without sacrificing low-level spatial details.
• Propose a two-pathway architecture that separates spatial details from semantic context.
• Design an efficient fusion mechanism to combine both pathways.
• Introduce a booster training strategy to improve accuracy without extra inference cost.
• Demonstrate effectiveness on Cityscapes, CamVid, and COCO-Stuff datasets.

Proposed method

• Detail Branch with wide channels and shallow layers to capture high-resolution spatial details.
• Semantic Branch with narrow channels and deep layers to capture high-level semantics using lightweight convolutions and fast downsampling.
• Context Embedding Block to enlarge receptive field in the Semantic Branch.
• Gather-and-Expansion (GE) Layer to build a lightweight yet expressive semantic path.
• Bilateral Guided Aggregation Layer to fuse Detail and Semantic Branch outputs guided by semantic context.
• Booster training with auxiliary prediction heads that improve accuracy during training but are discarded at inference.

Experimental results

Research questions

• RQ1 Can BiSeNet V2 achieve high segmentation accuracy while maintaining real-time inference speed?
• RQ2 Does separating spatial details and semantic context improve performance over single-path architectures at similar computation budgets?
• RQ3 How effective is the Bilateral Guided Aggregation Layer in fusing multi-scale details with semantics?
• RQ4 What is the impact of booster training on final performance without inference cost?

Key findings

• Achieves 72.6% mean IoU on Cityscapes test with 156 FPS on a GTX 1080 Ti.
• Detail and Semantic Branches provide complementary information, with fusion via Bilateral Guided Aggregation Layer yielding superior results to simple summation or concatenation.
• Semantic Branch can be lightweight yet effective due to depthwise convolutions and fast downsampling, while Detail Branch preserves spatial details.
• Booster training improves accuracy without increasing inference cost.
• Validated effectiveness on Cityscapes, CamVid, and COCO-Stuff datasets.