[论文解读] BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation

Changqian Yu, Changxin Gao+4 more|arXiv (Cornell University)|Apr 5, 2020

Advanced Neural Network Applications参考文献 66被引用 115

一句话总结

BiSeNet V2 引入了一个两路径架构（Detail Branch 用于空间细节，Semantic Branch 用于语义信息），配合 Bilateral Guided Aggregation Layer 的引导聚合以及 booster 训练，以在实时语义分割中实现高精度，例如 Cityscapes 测试集的 72.6% mIoU，156 FPS。

ABSTRACT

The low-level details and high-level semantics are both essential to the semantic segmentation task. However, to speed up the model inference, current approaches almost always sacrifice the low-level details, which leads to a considerable accuracy decrease. We propose to treat these spatial details and categorical semantics separately to achieve high accuracy and high efficiency for realtime semantic segmentation. To this end, we propose an efficient and effective architecture with a good trade-off between speed and accuracy, termed Bilateral Segmentation Network (BiSeNet V2). This architecture involves: (i) a Detail Branch, with wide channels and shallow layers to capture low-level details and generate high-resolution feature representation; (ii) a Semantic Branch, with narrow channels and deep layers to obtain high-level semantic context. The Semantic Branch is lightweight due to reducing the channel capacity and a fast-downsampling strategy. Furthermore, we design a Guided Aggregation Layer to enhance mutual connections and fuse both types of feature representation. Besides, a booster training strategy is designed to improve the segmentation performance without any extra inference cost. Extensive quantitative and qualitative evaluations demonstrate that the proposed architecture performs favourably against a few state-of-the-art real-time semantic segmentation approaches. Specifically, for a 2,048x1,024 input, we achieve 72.6% Mean IoU on the Cityscapes test set with a speed of 156 FPS on one NVIDIA GeForce GTX 1080 Ti card, which is significantly faster than existing methods, yet we achieve better segmentation accuracy.

研究动机与目标

• 在不牺牲低级空间细节的前提下推动实时语义分割。
• 提出一个两路径架构，将空间细节与语义上下文分离。
• 设计一个高效的融合机制，将两条路径结合起来。
• 引入 booster 训练策略，在不增加推理成本的情况下提升准确性。
• 在 Cityscapes、CamVid 和 COCO-Stuff 数据集上展示有效性。

提出的方法

• Detail Branch 采用宽通道和浅层，以捕捉高分辨率的空间细节。
• Semantic Branch 采用窄通道和深层来捕捉高层语义，使用轻量卷积和快速下采样。
• Context Embedding Block 用于在 Semantic Branch 中扩大感受野。
• Gather-and-Expansion (GE) Layer 构建一个轻量但具表达力的语义路径。
• Bilateral Guided Aggregation Layer 用于在语义上下文引导下融合 Detail 与 Semantic Branch 的输出。
• Booster training with auxiliary prediction heads that improve accuracy during training but are discarded at inference.

实验结果

研究问题

• RQ1 BiSeNet V2 是否在保持实时推理速度的同时实现高分割精度？
• RQ2 将空间细节与语义上下文分离是否在相似计算预算下提升了相对单路径架构的性能？
• RQ3 Bilateral Guided Aggregation Layer 在多尺度细节与语义融合中的有效性如何？
• RQ4 Booster 训练对最终性能的影响（不增加推理成本）如何？

主要发现

• 在 Cityscapes 测试上达到 72.6% 的 mean IoU，且 GTX 1080 Ti 上实现 156 FPS。
• Detail Branch 与 Semantic Branch 提供互补信息，通过 Bilateral Guided Aggregation Layer 的融合比简单求和或拼接得到更优结果。
• Semantic Branch 由于深度卷积和快速下采样可以保持轻量同时有效，而 Detail Branch 保留空间细节。
• Booster 训练在不增加推理成本的情况下提高准确性。
• 在 Cityscapes、CamVid 和 COCO-Stuff 数据集上验证了有效性。