[Paper Review] ISBNet: a 3D Point Cloud Instance Segmentation Network with Instance-aware Sampling and Box-aware Dynamic Convolution

Tuan Ngo, Binh‐Son Hua+1 more|arXiv (Cornell University)|Mar 1, 2023

3D Shape Modeling and Analysis4 citations

TL;DR

ISBNet proposes a cluster-free 3D point cloud instance segmentation network that uses Instance-aware Farthest Point Sampling (IA-FPS) to generate high-recall, discriminative instance kernels and Box-aware Dynamic Convolution to improve mask prediction. It achieves state-of-the-art performance on ScanNetV2 (55.9 AP), S3DIS (60.8 AP), and STPLS3D (49.2 AP), with fast inference at 237ms per scene.

ABSTRACT

Existing 3D instance segmentation methods are predominated by the bottom-up design -- manually fine-tuned algorithm to group points into clusters followed by a refinement network. However, by relying on the quality of the clusters, these methods generate susceptible results when (1) nearby objects with the same semantic class are packed together, or (2) large objects with loosely connected regions. To address these limitations, we introduce ISBNet, a novel cluster-free method that represents instances as kernels and decodes instance masks via dynamic convolution. To efficiently generate high-recall and discriminative kernels, we propose a simple strategy named Instance-aware Farthest Point Sampling to sample candidates and leverage the local aggregation layer inspired by PointNet++ to encode candidate features. Moreover, we show that predicting and leveraging the 3D axis-aligned bounding boxes in the dynamic convolution further boosts performance. Our method set new state-of-the-art results on ScanNetV2 (55.9), S3DIS (60.8), and STPLS3D (49.2) in terms of AP and retains fast inference time (237ms per scene on ScanNetV2). The source code and trained models are available at https://github.com/VinAIResearch/ISBNet.

Motivation & Objective

• To address the limitations of bottom-up 3D instance segmentation methods that rely on clustering, which often fail when objects are densely packed or large objects are loosely connected.
• To improve instance recall and feature discriminability by replacing centroid-based clustering with a novel instance-aware sampling strategy.
• To enhance mask prediction accuracy by leveraging 3D axis-aligned bounding boxes as an auxiliary geometric cue in dynamic convolution.
• To achieve state-of-the-art performance with minimal architectural changes and efficient inference across multiple benchmark datasets.

Proposed method

• Introduces Instance-aware Farthest Point Sampling (IA-FPS), which iteratively samples points based on predicted instance confidence to improve instance recall.
• Employs a Point Aggregator module that combines IA-FPS with a local aggregation layer to encode shape, size, and semantic features into instance-wise representations.
• Proposes Box-aware Dynamic Convolution, where predicted 3D axis-aligned bounding boxes are used as geometric priors to guide dynamic convolutional kernel generation.
• Uses a multi-head head to jointly predict instance masks, class labels, and 3D bounding boxes, with the box prediction serving as a geometric regularizer and inference cue.
• Adopts a lightweight dynamic convolution head with only 216 parameters, achieving strong performance while maintaining high inference speed.
• Implements a hierarchical sampling strategy in IA-FPS, splitting the sampling process into smaller chunks (e.g., 192, 128, 64) to improve stability and performance.

Experimental results

Research questions

• RQ1 Can instance-aware sampling improve instance recall and reduce misgrouping in 3D point cloud instance segmentation compared to traditional clustering-based methods?
• RQ2 Does incorporating 3D axis-aligned bounding boxes as an auxiliary supervision improve mask prediction accuracy in dynamic convolution-based 3D instance segmentation?
• RQ3 How does the design of dynamic convolution, particularly with geometric cues, affect performance and efficiency in 3D instance segmentation?
• RQ4 Can a cluster-free framework achieve state-of-the-art performance without extensive hyperparameter tuning per dataset?
• RQ5 What is the optimal trade-off between sampling strategy, network depth, and dynamic convolution architecture for high-accuracy and fast inference?

Key findings

• ISBNet achieves 55.9 AP on ScanNetV2, surpassing the previous SOTA by +2.7 AP, with a fast inference time of 237ms per scene.
• On S3DIS, ISBNet achieves 60.8 AP, a +2.4 AP improvement over the strongest baseline.
• On STPLS3D, ISBNet achieves 49.2 AP, a +3.0 AP gain over the previous state-of-the-art method.
• The ablation study shows that using 192, 128, and 64 as chunk sizes in IA-FPS yields the best performance (54.5 AP), outperforming single large chunks.
• The Box-aware Dynamic Convolution with only 216 parameters achieves 53.6 AP, demonstrating that geometric cues significantly boost performance even with minimal parameters.
• The runtime analysis confirms ISBNet is the fastest method among five SOTA approaches, with 152ms for backbone, 53ms for instance abstraction, and 32ms for mask decoding.