[Paper Review] MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation

Chen Liu, Yasutaka Furukawa|arXiv (Cornell University)|Feb 12, 2019

3D Shape Modeling and Analysis19 references81 citations

TL;DR

Introduces MASC, a sparse-convolution-based network that predicts semantic scores and multi-scale point affinities to cluster 3D points into instances without proposals, achieving state-of-the-art results on ScanNet.

ABSTRACT

We propose a new approach for 3D instance segmentation based on sparse convolution and point affinity prediction, which indicates the likelihood of two points belonging to the same instance. The proposed network, built upon submanifold sparse convolution [3], processes a voxelized point cloud and predicts semantic scores for each occupied voxel as well as the affinity between neighboring voxels at different scales. A simple yet effective clustering algorithm segments points into instances based on the predicted affinity and the mesh topology. The semantic for each instance is determined by the semantic prediction. Experiments show that our method outperforms the state-of-the-art instance segmentation methods by a large margin on the widely used ScanNet benchmark [2]. We share our code publicly at https://github.com/art-programmer/MASC.

Motivation & Objective

• Motivate robust 3D instance segmentation over large indoor scenes.
• Leverage sparse convolutions to process full-scene voxelized point clouds efficiently.
• Predict semantic scores and multi-scale voxel affinities to enable clustering into instances.
• Develop a simple clustering procedure that uses learned affinities and mesh topology.

Proposed method

• Use a U-Net with submanifold sparse convolutions to process the full scene voxel grid.
• Predict per-voxel semantic scores and multiple affinity branches at different scales between neighboring voxels.
• Define a clustering algorithm that merges nodes based on averaged multi-scale affinities and mesh topology.
• Assign each instance the semantic label with majority votes from its points.
• Augment data with random densification to improve local affinity prediction in sparse regions.

Experimental results

Research questions

• RQ1 Can multi-scale affinity prediction with sparse convolutions enable effective, proposal-free 3D instance segmentation?
• RQ2 How does the proposed clustering algorithm leverage predicted affinities and mesh topology to form coherent instances?
• RQ3 What is the impact of using two affinity scales on ScanNet performance?
• RQ4 How does MASC compare to state-of-the-art methods on the ScanNet benchmark?

Key findings

• Outperforms state-of-the-art methods on the ScanNet benchmark for 3D instance segmentation (AP at IoU 0.5).
• The proposed Ours method achieves higher scores across most categories compared to 3D-SIS, GSPN, and SGPN in Table 1.
• A clustering-based, multi-scale affinity approach yields strong instance segmentation without proposal-based detections.
• Semantic labels for instances are determined by the maximum votes from their constituent points.
• The method includes data densification and planar-object handling to improve segmentation of co-planar or flat objects.