[Paper Review] SQN: Weakly-Supervised Semantic Segmentation of Large-Scale 3D Point Clouds with 1000x Fewer Labels.

Qingyong Hu, Bo Yang+5 more|arXiv (Cornell University)|Apr 11, 2021

3D Shape Modeling and Analysis22 references18 citations

TL;DR

This paper proposes the Semantic Query Network (SQN), a weakly supervised method for semantic segmentation of large-scale 3D point clouds that achieves state-of-the-art performance using only 0.1% of the labeled points required by fully supervised methods. By leveraging semantic similarity between neighboring points to implicitly augment supervision, SQN reduces labeling effort by 1000x while maintaining high accuracy across six large-scale datasets.

ABSTRACT

We study the problem of labelling effort for semantic segmentation of large-scale 3D point clouds. Existing works usually rely on densely annotated point-level semantic labels to provide supervision for network training. However, in real-world scenarios that contain billions of points, it is impractical and extremely costly to manually annotate every single point. In this paper, we first investigate whether dense 3D labels are truly required for learning meaningful semantic representations. Interestingly, we find that the segmentation performance of existing works only drops slightly given as few as 1% of the annotations. However, beyond this point (e.g. 1 per thousand and below) existing techniques fail catastrophically. To this end, we propose a new weak supervision method to implicitly augment the total amount of available supervision signals, by leveraging the semantic similarity between neighboring points. Extensive experiments demonstrate that the proposed Semantic Query Network (SQN) achieves state-of-the-art performance on six large-scale open datasets under weak supervision schemes, while requiring only 1000x fewer labeled points for training. The code is available at this https URL.

Motivation & Objective

• To investigate whether dense point-level annotations are truly necessary for effective 3D semantic segmentation.
• To reduce the prohibitive labeling cost of large-scale 3D point clouds, which can contain billions of points.
• To develop a weakly supervised method that maintains high segmentation performance with drastically reduced annotation requirements.
• To explore whether semantic similarity between neighboring points can serve as a proxy for supervision to boost model learning.

Proposed method

• SQN introduces a novel weak supervision paradigm that leverages local semantic similarity between points to implicitly increase the effective supervision signal.
• The method uses a learnable query mechanism to identify and exploit semantically coherent neighborhoods in the point cloud.
• It formulates a contrastive learning objective that encourages feature embeddings of semantically similar neighboring points to be close in the embedding space.
• The network is trained end-to-end with a combination of sparse point-level annotations and self-supervised contrastive signals from neighboring points.
• The framework is designed to be scalable and efficient, suitable for point clouds with billions of points.

Experimental results

Research questions

• RQ1 Can semantic segmentation performance be maintained with only 1% of the standard number of labeled points?
• RQ2 What is the performance drop-off threshold when label density is reduced below 1%?
• RQ3 Can semantic similarity between neighboring points be effectively used to augment weak supervision in 3D segmentation?
• RQ4 Does the proposed method achieve state-of-the-art performance under weak supervision with minimal labeling?

Key findings

• SQN achieves state-of-the-art performance on six large-scale 3D point cloud datasets under weak supervision settings.
• The method maintains high segmentation accuracy even when only 0.1% of the points are labeled, representing a 1000x reduction in labeling effort.
• Performance degrades significantly when label density drops below 1%, indicating a critical threshold for existing methods.
• The use of semantic similarity between neighboring points enables effective self-supervision, significantly improving model generalization with minimal annotations.