[Paper Review] PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation

Li Jiang, Hengshuang Zhao+4 more|arXiv (Cornell University)|Apr 3, 2020

3D Shape Modeling and AnalysisEngineering57 references32 citations

TL;DR

PointGroup introduces a bottom-up 3D instance segmentation framework that uses semantic labels and per-point offsets to group points with dual coordinate sets (original and offset-shifted) and a ScoreNet to select high-quality clusters, achieving state-of-the-art results on ScanNet v2 and S3DIS.

ABSTRACT

Instance segmentation is an important task for scene understanding. Compared to the fully-developed 2D, 3D instance segmentation for point clouds have much room to improve. In this paper, we present PointGroup, a new end-to-end bottom-up architecture, specifically focused on better grouping the points by exploring the void space between objects. We design a two-branch network to extract point features and predict semantic labels and offsets, for shifting each point towards its respective instance centroid. A clustering component is followed to utilize both the original and offset-shifted point coordinate sets, taking advantage of their complementary strength. Further, we formulate the ScoreNet to evaluate the candidate instances, followed by the Non-Maximum Suppression (NMS) to remove duplicates. We conduct extensive experiments on two challenging datasets, ScanNet v2 and S3DIS, on which our method achieves the highest performance, 63.6% and 64.0%, compared to 54.9% and 54.4% achieved by former best solutions in terms of mAP with IoU threshold 0.5.

Motivation & Objective

• Motivate accurate 3D instance segmentation by leveraging void space between objects in point clouds.
• Develop a bottom-up architecture that jointly predicts per-point semantic labels and centroid-oriented offsets.
• Propose dual-set clustering (original and offset-shifted points) to improve instance separation, especially for closely spaced objects.
• Introduce ScoreNet to evaluate and select high-quality clusters prior to NMS.
• Demonstrate state-of-the-art performance on ScanNet v2 and S3DIS datasets.

Proposed method

• A two-branch backbone generates per-point semantic labels and centroid offsets.
• Points are shifted toward their instance centroids using predicted offsets to form a shifted coordinate set.
• Clustering is performed on both the original (P) and shifted (Q) coordinate sets to form C^p and C^q, then merged into C = C^p ∪ C^q.
• ScoreNet encodes cluster features and outputs a score for each cluster proposal.
• ScoreNet scores are trained with a soft IoU-based target to reflect cluster quality, guiding NMS for final predictions.
• End-to-end training optimizing L = L_sem + L_o_dir + L_o_reg + L_c_score.

Experimental results

Research questions

• RQ1 How can 3D point cloud instance segmentation be improved by leveraging the void space between objects?
• RQ2 Can a dual-coordinate clustering approach (original and offset-shifted points) outperform single-coordinate clustering for separating adjacent instances?
• RQ3 Does a learned cluster-scoring module (ScoreNet) improve selection of high-quality instance proposals for NMS?
• RQ4 What is the impact of combining original and shifted coordinate clustering on standard 3D instance segmentation benchmarks?

Key findings

• PointGroup achieves 63.6% mAP50 on ScanNet v2 test, outperforming prior best by 8.7 percentage points.
• On S3DIS, PointGroup attains 64.0% mAP50, 69.6% mPrec50, and 69.2% mRec50, surpassing prior methods by large margins.
• Ablation shows dual-set clustering (P and Q) yields best performance over P-only or Q-only variants.
• Clustering radius r and ScoreNet contribute significantly to performance; using both coordinate sets with ScoreNet provides the best results.
• PointGroup ranks first in the majority of classes on ScanNet v2 and achieves state-of-the-art results on S3DIS across multiple metrics.