[Paper Review] Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes

Julian Chibane, Francis Engelmann+2 more|arXiv (Cornell University)|Jun 2, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Box2Mask, a weakly supervised method for 3D semantic instance segmentation that trains deep models using only 3D bounding box annotations, avoiding costly per-point labeling. Inspired by Hough voting, the method directly regresses box parameters and clusters votes to generate dense instance masks, achieving 97% of the mAP50 score of fully supervised models on ScanNet and enabling strong performance on the ARKitScenes dataset with only box-level annotations.

ABSTRACT

Current 3D segmentation methods heavily rely on large-scale point-cloud datasets, which are notoriously laborious to annotate. Few attempts have been made to circumvent the need for dense per-point annotations. In this work, we look at weakly-supervised 3D semantic instance segmentation. The key idea is to leverage 3D bounding box labels which are easier and faster to annotate. Indeed, we show that it is possible to train dense segmentation models using only bounding box labels. At the core of our method, ame{}, lies a deep model, inspired by classical Hough voting, that directly votes for bounding box parameters, and a clustering method specifically tailored to bounding box votes. This goes beyond commonly used center votes, which would not fully exploit the bounding box annotations. On ScanNet test, our weakly supervised model attains leading performance among other weakly supervised approaches (+18 mAP@50). Remarkably, it also achieves 97% of the mAP@50 score of current fully supervised models. To further illustrate the practicality of our work, we train Box2Mask on the recently released ARKitScenes dataset which is annotated with 3D bounding boxes only, and show, for the first time, compelling 3D instance segmentation masks.

Motivation & Objective

• To reduce the annotation burden in 3D semantic instance segmentation by replacing dense per-point annotations with easier-to-annotate 3D bounding boxes.
• To develop a deep learning method that fully exploits bounding box annotations for dense prediction, going beyond simple center-point supervision.
• To demonstrate that weakly supervised models trained on bounding boxes can match or exceed the performance of prior weakly supervised methods.
• To enable high-quality 3D instance segmentation on large-scale datasets like ARKitScenes, which are annotated only with 3D bounding boxes.

Proposed method

• A deep Hough voting network is proposed that directly regresses 3D bounding box parameters (center, size, class) from point cloud features.
• The method uses a differentiable clustering mechanism tailored to vote clusters from bounding box predictions to generate instance masks.
• A novel point-to-box association function assigns points to ground truth boxes, with undecided points (in multiple boxes) supervised using heuristics like closest or smallest box.
• The model is trained end-to-end using a weakly supervised loss that leverages the point-to-box associations to supervise dense predictions.
• The framework is adaptable to both weakly supervised and fully supervised settings by switching the supervision signal.
• The method outperforms a baseline detection-only approach that uses post-processing for mask generation.

Experimental results

Research questions

• RQ1 Can 3D bounding box annotations alone be sufficient to train a high-performance 3D semantic instance segmentation model?
• RQ2 How does the performance of a weakly supervised model trained on bounding boxes compare to fully supervised state-of-the-art models?
• RQ3 Can the proposed method generalize beyond simple point associations to learn complete object priors?
• RQ4 How robust is the method to noisy or incomplete bounding box annotations?
• RQ5 Can the method produce compelling instance segmentation results on large-scale datasets annotated only with 3D bounding boxes?

Key findings

• On the ScanNet test split, Box2Mask achieves 64.7 mAP50, which is 97% of the mAP50 score of the fully supervised SSTNet model (64.3 mAP50).
• The model outperforms previous weakly supervised methods by a large margin, improving mAP50 by +18 over prior weakly supervised approaches.
• Even when only 80% of the bounding boxes are present (10% missing), the model maintains strong performance, with only a ~4 mAP50 drop.
• When trained on noisy box labels with up to 20 cm error in corner positions, the model shows robustness with minimal performance degradation.
• On the ARKitScenes dataset—annotated exclusively with 3D bounding boxes—Box2Mask produces the first compelling 3D instance segmentation masks, demonstrating practical applicability.
• The method significantly outperforms a detection-only baseline (+11.8 mAP50 on ScanNet), showing that it learns richer object priors than simple box-to-point association.