[Paper Review] Scalable SoftGroup for 3D Instance Segmentation on Point Clouds

Thang X. Vu, Kookhoi Kim+4 more|arXiv (Cornell University)|Sep 17, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes SoftGroup and SoftGroup++, a scalable framework for 3D instance segmentation on point clouds that improves accuracy by using soft semantic scores instead of hard predictions to reduce error propagation during grouping. SoftGroup++ achieves up to 6× faster inference on large-scale scenes via octree k-NN, class-aware pyramid scaling, and late devoxelization, outperforming prior methods by 6–16% in AP50 across indoor and outdoor benchmarks.

ABSTRACT

This paper considers a network referred to as SoftGroup for accurate and scalable 3D instance segmentation. Existing state-of-the-art methods produce hard semantic predictions followed by grouping instance segmentation results. Unfortunately, errors stemming from hard decisions propagate into the grouping, resulting in poor overlap between predicted instances and ground truth and substantial false positives. To address the abovementioned problems, SoftGroup allows each point to be associated with multiple classes to mitigate the uncertainty stemming from semantic prediction. It also suppresses false positive instances by learning to categorize them as background. Regarding scalability, the existing fast methods require computational time on the order of tens of seconds on large-scale scenes, which is unsatisfactory and far from applicable for real-time. Our finding is that the $k$-Nearest Neighbor ($k$-NN) module, which serves as the prerequisite of grouping, introduces a computational bottleneck. SoftGroup is extended to resolve this computational bottleneck, referred to as SoftGroup++. The proposed SoftGroup++ reduces time complexity with octree $k$-NN and reduces search space with class-aware pyramid scaling and late devoxelization. Experimental results on various indoor and outdoor datasets demonstrate the efficacy and generality of the proposed SoftGroup and SoftGroup++. Their performances surpass the best-performing baseline by a large margin (6\% $\sim$ 16\%) in terms of AP$_{50}$. On datasets with large-scale scenes, SoftGroup++ achieves a 6$ imes$ speed boost on average compared to SoftGroup. Furthermore, SoftGroup can be extended to perform object detection and panoptic segmentation with nontrivial improvements over existing methods. The source code and trained models are available at \url{https://github.com/thangvubk/SoftGroup}.

Motivation & Objective

• Address error propagation from hard semantic predictions in 3D instance segmentation, which leads to low IoU and false positives.
• Improve instance segmentation accuracy by allowing points to belong to multiple classes via soft semantic scores.
• Enable real-time inference on large-scale 3D scenes by reducing computational bottlenecks in k-NN and search space.
• Extend the method to object detection and panoptic segmentation with consistent performance gains.
• Develop a scalable, generalizable framework that maintains high accuracy while reducing inference time on massive point clouds.

Proposed method

• SoftGroup performs instance grouping using soft semantic scores instead of hard one-hot predictions, reducing misclassification from local ambiguity.
• A top-down refinement stage classifies instance proposals as positive or negative based on IoU with ground truth, refining positives and suppressing false positives.
• SoftGroup++ replaces vanilla k-NN with octree-based k-NN to reduce time complexity from O(n²) to O(n log n).
• Class-aware pyramid scaling reduces the search space by focusing on high-confidence, class-specific regions during grouping.
• Late devoxelization delays voxelization until later stages, preserving spatial resolution and reducing intermediate computation.
• The framework supports end-to-end training and inference for instance segmentation, object detection, and panoptic segmentation.

Experimental results

Research questions

• RQ1 Can soft semantic scores improve instance segmentation accuracy by mitigating error propagation from noisy semantic predictions?
• RQ2 How can the computational bottleneck of k-NN in large-scale 3D point cloud processing be effectively reduced?
• RQ3 To what extent can the proposed method scale to large-scale scenes while maintaining high accuracy?
• RQ4 Can the SoftGroup framework be generalized to other 3D perception tasks like object detection and panoptic segmentation?
• RQ5 What is the impact of back-fusion of instance masks on semantic segmentation performance?

Key findings

• SoftGroup improves AP50 by 6–16% over the best baseline on multiple indoor and outdoor datasets, with gains up to 16% on ScanNet.
• SoftGroup++ achieves a 6× average speedup on large-scale scenes compared to SoftGroup, reducing inference time from tens of seconds to under 10 seconds for 4.5M-point scenes.
• On the ScanNet v2 validation set, back-fusion of instance masks into semantic maps improves mIoU by 0.7 points, especially benefiting semantically similar classes like cabinet and fridge.
• SoftGroup and SoftGroup++ achieve AP50 of 59.6 and AP25 of 71.7 on object detection, outperforming existing methods on ScanNet.
• In panoptic segmentation on SemanticKITTI, SoftGroup and SoftGroup++ achieve competitive PQ scores, with potential for further gains when combined with advanced models like Panoptic-PHNet.
• The integration of SoftGroup with SSTNet boosts AP50 from 64.9% to 67.7%, demonstrating strong synergy with superpoint-based methods.