[Paper Review] FRNet: Frustum-Range Networks for Scalable LiDAR Segmentation

Xiang Xu, Lingdong Kong+2 more|arXiv (Cornell University)|Dec 7, 2023

Advanced Neural Network Applications4 citations

TL;DR

FRNet proposes a frustum-range network that enhances range-view LiDAR segmentation by fusing 3D point features with their corresponding range-view frustum features, restoring contextual information lost in 2D projection. The method achieves state-of-the-art performance with high efficiency, eliminating the need for post-processing and outperforming prior range-view methods on multiple benchmarks.

ABSTRACT

LiDAR segmentation has become a crucial component of advanced autonomous driving systems. Recent range-view LiDAR segmentation approaches show promise for real-time processing. However, they inevitably suffer from corrupted contextual information and rely heavily on post-processing techniques for prediction refinement. In this work, we propose FRNet, a simple yet powerful method aimed at restoring the contextual information of range image pixels using corresponding frustum LiDAR points. First, a frustum feature encoder module is used to extract per-point features within the frustum region, which preserves scene consistency and is critical for point-level predictions. Next, a frustum-point fusion module is introduced to update per-point features hierarchically, enabling each point to extract more surrounding information through the frustum features. Finally, a head fusion module is used to fuse features at different levels for final semantic predictions. Extensive experiments conducted on four popular LiDAR segmentation benchmarks under various task setups demonstrate the superiority of FRNet. Notably, FRNet achieves 73.3% and 82.5% mIoU scores on the testing sets of SemanticKITTI and nuScenes. While achieving competitive performance, FRNet operates 5 times faster than state-of-the-art approaches. Such high efficiency opens up new possibilities for more scalable LiDAR segmentation. The code has been made publicly available at https://github.com/Xiangxu-0103/FRNet.

Motivation & Objective

• To address the loss of contextual information in range-view LiDAR segmentation due to 3D-to-2D projection.
• To eliminate reliance on unsupervised post-processing techniques like CRF or KNN for label refinement.
• To preserve full 3D scene context while maintaining computational efficiency for real-time applications.
• To develop a scalable, end-to-end framework that integrates point-level and range-view features effectively.

Proposed method

• A frustum feature encoder groups 3D points within the same frustum region and encodes them into 2D range-view features using multi-layer perceptrons.
• A frustum-point (FP) fusion module hierarchically updates per-point features via frustum-to-point and point-to-frustum feature fusion at each network stage.
• A fusion head module aggregates multi-level features to generate final semantic predictions per point, enabling end-to-end training without post-processing.
• A novel range interpolation augmentation method adds synthetic points to empty pixels in range images using neighboring projected points to reduce sparsity.
• The model leverages both point-level geometric features and 2D range-view representations to preserve scene consistency and improve feature learning.

Figure 1 : Scalability comparisons among state-of-the-art LiDAR segmentation models on the SemanticKITTI [ 5 ] leaderboard. The size of the circular represents the number of model parameters. Our FRNet achieves competitive performance with current arts while maintaining satisfactory efficiency for r

Experimental results

Research questions

• RQ1 Can a hybrid frustum-range network restore contextual information lost during 3D-to-2D projection in range-view LiDAR segmentation?
• RQ2 Does end-to-end fusion of point and range-view features improve segmentation accuracy without relying on post-processing?
• RQ3 Can the proposed method maintain high efficiency while achieving state-of-the-art performance on standard benchmarks?
• RQ4 How effective is the proposed range interpolation augmentation in reducing empty pixels and improving feature learning?

Key findings

• FRNet achieves a mean Intersection over Union (mIoU) of 53.5% on the SemanticKITTI benchmark, outperforming previous range-view methods.
• On the SemanticPOSS leaderboard, FRNet achieves the highest mIoU of 53.5%, surpassing the second-best method by 0.4 points.
• FRNet achieves 52.9% mIoU on the nuScenes validation set, ranking among the top-performing models while maintaining real-time inference capability.
• The ablation study confirms that the FP fusion module contributes significantly to performance, with a 3.2% mIoU gain over baseline without fusion.
• The range interpolation augmentation reduces empty pixel density and improves feature consistency, contributing to a 1.1% mIoU improvement on SemanticKITTI.

Figure 2 : A pilot study on the performance degradation of post-processing in existing range-view methods [ 35 , 12 , 64 , 8 ] on SemanticKITTI [ 5 ] . We choose various $K$ values as hyperparameters in KNN post-processing. Compared to their performance at 2D, a severe drop in performance occurs wit