[Paper Review] GFNet: Geometric Flow Network for 3D Point Cloud Semantic Segmentation

Haibo Qiu, Baosheng Yu+1 more|arXiv (Cornell University)|Jul 6, 2022

3D Surveying and Cultural Heritage25 citations

TL;DR

GFNet learns bidirectional geometric flow between range-view and BEV projections to fuse multi-view features for improved 3D point cloud semantic segmentation, achieving state-of-the-art results among projection-based models on SemanticKITTI and nuScenes.

ABSTRACT

Point cloud semantic segmentation from projected views, such as range-view (RV) and bird's-eye-view (BEV), has been intensively investigated. Different views capture different information of point clouds and thus are complementary to each other. However, recent projection-based methods for point cloud semantic segmentation usually utilize a vanilla late fusion strategy for the predictions of different views, failing to explore the complementary information from a geometric perspective during the representation learning. In this paper, we introduce a geometric flow network (GFNet) to explore the geometric correspondence between different views in an align-before-fuse manner. Specifically, we devise a novel geometric flow module (GFM) to bidirectionally align and propagate the complementary information across different views according to geometric relationships under the end-to-end learning scheme. We perform extensive experiments on two widely used benchmark datasets, SemanticKITTI and nuScenes, to demonstrate the effectiveness of our GFNet for project-based point cloud semantic segmentation. Concretely, GFNet not only significantly boosts the performance of each individual view but also achieves state-of-the-art results over all existing projection-based models. Code is available at \url{https://github.com/haibo-qiu/GFNet}.

Motivation & Objective

• Motivate improving projection-based point cloud segmentation by exploiting geometric correspondences between RV and BEV rather than relying on vanilla late fusion.
• Propose GFNet with a geometric flow module to bidirectionally propagate information between RV and BEV in an end-to-end framework.
• Enable end-to-end training by replacing KNN post-processing with KPConv in a two-branch RV/BEV architecture.
• Demonstrate effectiveness on large-scale benchmarks SemanticKITTI and nuScenes and achieve state-of-the-art results among projection-based models.

Proposed method

• Two-branch network architecture processes RV and BEV inputs with encoder-decoder backbones.
• Geometric Flow Module (GFM) performs geometric alignment across RV and BEV using their transformation between views.
• GFM includes an attention fusion step that combines aligned features with target features via self-attention and residual connections.
• Geometric alignment uses the original point cloud as a bridge to compute cross-view transformation matrices.
• KPConv is used on top of GFNet to replace KNN, enabling end-to-end trainability.
• Loss combines 2D and 3D supervision with Lovasz-Softmax and cross-entropy terms to train all components end-to-end.

Experimental results

Research questions

• RQ1 Can geometric correspondences between RV and BEV be leveraged to improve cross-view information propagation for point cloud segmentation?
• RQ2 Does bidirectional geometric flow between RV and BEV improve per-view representations and overall fusion compared to vanilla late fusion?
• RQ3 What is the impact of attention-based fusion in the GFM on segmentation performance?
• RQ4 How does GFNet perform on large-scale benchmarks SemanticKITTI and nuScenes compared to existing projection-based methods?
• RQ5 Is end-to-end training with KPConv effective for multi-view projection-based segmentation?

Key findings

• GFNet achieves improved mIoU over all compared projection-based models on SemanticKITTI validation.
• Both RV-Single and BEV-Single branches gain significant performance from incorporating GFM, with substantial gains when allowing flow between views.
• RV-Flow and BEV-Flow demonstrate strong cross-view improvements, and concatenation with KPConv yields the best GFNet performance.
• Attention in GFM (softmax) provides marginal gains over sigmoid and improves fusion effectiveness.
• Ablations show that jointly training with 2D and 3D supervision (λ configuration) yields best results, with end-to-end optimization enhancing performance.