[论文解读] BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud

Mong H. Ng, Kaahan Radia+4 more|arXiv (Cornell University)|Jun 19, 2020

Advanced Neural Network Applications参考文献 30被引用 37

一句话总结

BEV-Seg 引入一个两阶段管线，使用单目深度和语义分割来构建一个语义点云，再将其投影至 BEV 以实现最终分割，达到最先进的结果并具备更好的迁移性。

ABSTRACT

Bird's-eye-view (BEV) is a powerful and widely adopted representation for road scenes that captures surrounding objects and their spatial locations, along with overall context in the scene. In this work, we focus on bird's eye semantic segmentation, a task that predicts pixel-wise semantic segmentation in BEV from side RGB images. This task is made possible by simulators such as Carla, which allow for cheap data collection, arbitrary camera placements, and supervision in ways otherwise not possible in the real world. There are two main challenges to this task: the view transformation from side view to bird's eye view, as well as transfer learning to unseen domains. Existing work transforms between views through fully connected layers and transfer learns via GANs. This suffers from a lack of depth reasoning and performance degradation across domains. Our novel 2-staged perception pipeline explicitly predicts pixel depths and combines them with pixel semantics in an efficient manner, allowing the model to leverage depth information to infer objects' spatial locations in the BEV. In addition, we transfer learning by abstracting high-level geometric features and predicting an intermediate representation that is common across different domains. We publish a new dataset called BEVSEG-Carla and show that our approach improves state-of-the-art by 24% mIoU and performs well when transferred to a new domain.

研究动机与目标

• Motivate robust BEV semantic segmentation from RGB cameras without LiDAR.
• Leverage explicit depth reasoning and geometry to transform side views to BEV.
• Improve transfer learning across domains by abstracting a common intermediate representation.
• Provide a CARLA-based dataset with diverse weather and environments.
• Demonstrate state-of-the-art performance and transfer gains over prior methods.

提出的方法

• Two-stage pipeline: stage 1 creates a semantic point cloud by fusing side-view semantic maps and monocular depth for multiple views using pin-hole camera geometry.
• Project the semantic point cloud to an incomplete BEV through orthographic projection with height-based conflict resolution.
• Stage 2 uses a parser network to transform the incomplete BEV into a full BEV semantic segmentation via an expanded one-hot representation.
• Depth and segmentation modules are trained with ground-truths in the side view (depth from LiDAR projection, segmentation labels).
• Stage 2 operates on a common intermediate representation to improve transferability across domains.

实验结果

研究问题

• RQ1 Can explicit depth reasoning and geometric projection improve BEV semantic segmentation from RGB images?
• RQ2 Does a modular two-stage pipeline enhance transfer learning across different CARLA environments compared to end-to-end methods?
• RQ3 What is the impact of using an intermediate representation on BEV segmentation quality and domain transfer?
• RQ4 How does BEV-Seg perform on a newly released BEV dataset (BEVSEG-Carla) relative to prior methods?

主要发现

• BEV-Seg improves mIoU from 36.4% (VPN baseline) to 60.4% on BEVSEG-Carla in the source domain.
• Transfer from clear noon to wet sunset yields 44.5% mIoU for BEV-Seg versus 27.8% for the VPN baseline.
• Per-class IoU shows BEV-Seg better captures pedestrians, road lines, lanes, signs, and smaller objects compared to VPN.
• Oracle variants (ground truth depth/segmentation) indicate depth and segmentation accuracy are critical for peak BEV performance, with depth oracle reaching 66.5% and combined depth+segmentation oracle at 67.3%.
• The modular intermediate representation significantly reduces domain gap, enabling effective transfer without retraining stage 2.