[Paper Review] Learning Gaussian Instance Segmentation in Point Clouds

Shih‐Hung Liu, Shangyi Yu+3 more|arXiv (Cornell University)|Jul 20, 2020

3D Shape Modeling and Analysis36 references36 citations

TL;DR

GICN learns Gaussian center heatmaps to locate instance centers in 3D point clouds, then predicts sizes, boxes, and masks in a single-stage, anchor-free pipeline that achieves state-of-the-art on ScanNet and S3DIS.

ABSTRACT

This paper presents a novel method for instance segmentation of 3D point clouds. The proposed method is called Gaussian Instance Center Network (GICN), which can approximate the distributions of instance centers scattered in the whole scene as Gaussian center heatmaps. Based on the predicted heatmaps, a small number of center candidates can be easily selected for the subsequent predictions with efficiency, including i) predicting the instance size of each center to decide a range for extracting features, ii) generating bounding boxes for centers, and iii) producing the final instance masks. GICN is a single-stage, anchor-free, and end-to-end architecture that is easy to train and efficient to perform inference. Benefited from the center-dictated mechanism with adaptive instance size selection, our method achieves state-of-the-art performance in the task of 3D instance segmentation on ScanNet and S3DIS datasets.

Motivation & Objective

• Motivate 3D instance segmentation directly on point clouds without predefined anchors or box proposals.
• Introduce a center-dedicated, size-aware framework to predict instance centers, sizes, boxes, and masks.
• Leverage Gaussian center heatmaps to enable intuitive visualization and efficient center selection.
• Enable end-to-end training for improved training stability and inference efficiency.

Proposed method

• Predict Gaussian center heatmaps indicating instance centers over the whole scene.
• Select a small set of center candidates using a center selection mechanism that avoids redundant centers.
• Predict instance sizes for centers to determine adaptive neighborhoods for feature extraction.
• Predict 3D bounding boxes for each center using a size-aware context and a shared backbone.
• Predict instance masks aligned with the predicted bounding boxes using a joint mask network.
• Train with a multi-term loss including center, size, IoU (GIoU), and mask losses.

Experimental results

Research questions

• RQ1 Can Gaussian center heatmaps effectively represent instance centers in 3D point clouds?
• RQ2 Does a center-dictated, size-aware pipeline improve 3D instance segmentation over anchor- or proposal-based methods?
• RQ3 How does center selection interact with bounding box and mask prediction to produce accurate instances?
• RQ4 What gains in accuracy and efficiency arise from size-aware neighborhood extraction and end-to-end training?

Key findings

• GICN achieves 68.5% mPrec and 50.8% mRec on S3DIS with IoU 0.5, outperforming ASIS, 3D-BoNet, and 3D-BEVIS.
• On ScanNet v2, GICN attains the highest mean AP@50% across published methods at the time of submission.
• Center heatmaps closely approximate ground-truth center distributions and enable effective center candidate selection.
• The size-aware bounding-box prediction and end-to-end training contribute to strong 3D instance segmentation performance.
• Ablation shows significant drops when removing center prediction, focal loss, or semantic radius priors, validating their importance.