[Paper Review] (AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network

Ran Cheng, Ryan Razani+3 more|arXiv (Cornell University)|Feb 8, 2021

3D Shape Modeling and Analysis39 references27 citations

TL;DR

The paper proposes (AF)²-S3Net, an end-to-end 3D sparse CNN with Attentive Feature Fusion (AF2M) and Adaptive Feature Selection (AFSM) modules to improve LiDAR semantic segmentation, achieving state-of-the-art on SemanticKITTI and strong generalization to nuScenes-lidarseg.

ABSTRACT

Autonomous robotic systems and self driving cars rely on accurate perception of their surroundings as the safety of the passengers and pedestrians is the top priority. Semantic segmentation is one the essential components of environmental perception that provides semantic information of the scene. Recently, several methods have been introduced for 3D LiDAR semantic segmentation. While, they can lead to improved performance, they are either afflicted by high computational complexity, therefore are inefficient, or lack fine details of smaller instances. To alleviate this problem, we propose AF2-S3Net, an end-to-end encoder-decoder CNN network for 3D LiDAR semantic segmentation. We present a novel multi-branch attentive feature fusion module in the encoder and a unique adaptive feature selection module with feature map re-weighting in the decoder. Our AF2-S3Net fuses the voxel based learning and point-based learning into a single framework to effectively process the large 3D scene. Our experimental results show that the proposed method outperforms the state-of-the-art approaches on the large-scale SemanticKITTI benchmark, ranking 1st on the competitive public leaderboard competition upon publication.

Motivation & Objective

• Motivate accurate 3D LiDAR semantic segmentation for autonomous systems with sparse point clouds.
• Introduce a dual-attention architecture to fuse multi-scale features and emphasize fine details.
• Offer a unified voxel-point processing framework that blends voxel-based and point-based learning for large-scale scenes.
• Demonstrate state-of-the-art performance on SemanticKITTI and validate generalization on nuScenes-lidarseg and ModelNet40.

Proposed method

• Propose AF2M to fuse multi-branch features across small, medium, and large kernels representing point-based and voxel-based contexts.
• Introduce AFSM in the decoder to adaptively re-weight and select multi-scale features.
• Build end-to-end 3D sparse CNN on Minkowski Engine with a residual backbone.
• Use a composite loss combining Exponential-Log, geo-aware anisotropic, and Lovász losses (weights: 1, 1.5, 1.5 respectively).
• Process LiDAR frames as sparse tensors with coordinates C and features F (x,y,z, intensity, normals) for per-point segmentation.

Experimental results

Research questions

• RQ1 Can AF2M capture both local fine details and global context in sparse 3D LiDAR data?
• RQ2 Does the Adaptive Feature Selection Module improve generalization by re-weighting multi-scale features in the decoder?
• RQ3 How does (AF)²-S3Net perform on SemanticKITTI, nuScenes-lidarseg, and ModelNet40 compared to state-of-the-art?
• RQ4 What is the impact of the combined loss (Exponential-Log, geo-aware anisotropic, Lovász) on segmentation accuracy?

Key findings

• (AF)²-S3Net achieves state-of-the-art mean IoU (69.7) on SemanticKITTI test set, outperforming SPVNAS and MinkNet42 baselines.
• The method shows strong per-class gains, notably in small object classes like Bicycle (65.4), Motorcycle (86.8), and Pole (61.5).
• On nuScenes-lidarseg validation, (AF)²-S3Net attains Mean IoU of 62.2 and FW IoU of 83.0, surpassing MinkNet42 and SalsaNext baselines.
• Ablation studies show AF2M and AFSM together yield 68.6 mIoU, and with Lovász and geo-aware losses reach 74.2 mIoU on SemanticKITTI validation.
• ModelNet40 classification with AF2M achieves 93.16% overall accuracy, comparable to leading point-based methods.
• Qualitative results indicate improved fine-detail capture (e.g., cars, vegetation) and better distance-based performance.