[Paper Review] Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts

Ji Hou, Benjamin Graham+2 more|arXiv (Cornell University)|Dec 16, 2020

3D Surveying and Cultural HeritageEarth and Planetary Sciences70 references17 citations

TL;DR

This paper proposes Contrastive Scene Contexts, a 3D self-supervised pre-training method that enhances data efficiency in 3D scene understanding by jointly leveraging point-level correspondence and spatial context. By pre-training on unannotated point clouds and fine-tuning with minimal labels—achieving 89% of full-annotation performance on instance segmentation with only 0.1% labeled points—this approach significantly reduces annotation costs while outperforming prior methods like PointContrast.

ABSTRACT

The rapid progress in 3D scene understanding has come with growing demand for data; however, collecting and annotating 3D scenes (e.g. point clouds) are notoriously hard. For example, the number of scenes (e.g. indoor rooms) that can be accessed and scanned might be limited; even given sufficient data, acquiring 3D labels (e.g. instance masks) requires intensive human labor. In this paper, we explore data-efficient learning for 3D point cloud. As a first step towards this direction, we propose Contrastive Scene Contexts, a 3D pre-training method that makes use of both point-level correspondences and spatial contexts in a scene. Our method achieves state-of-the-art results on a suite of benchmarks where training data or labels are scarce. Our study reveals that exhaustive labelling of 3D point clouds might be unnecessary; and remarkably, on ScanNet, even using 0.1% of point labels, we still achieve 89% (instance segmentation) and 96% (semantic segmentation) of the baseline performance that uses full annotations.

Motivation & Objective

• To address the high cost and scarcity of annotated 3D point clouds in real-world scene understanding.
• To explore data-efficient 3D learning by reducing reliance on extensive human-annotated data.
• To improve transferability and performance of 3D models on downstream tasks like instance segmentation and object detection with limited supervision.
• To enable practical data collection strategies such as active labeling through learned feature clustering.
• To demonstrate that exhaustive 3D labeling may be unnecessary when using context-aware self-supervised pre-training.

Proposed method

• Proposes a contrastive learning framework that jointly optimizes for point-level correspondence and spatial context in 3D scenes.
• Uses data augmentations such as random cropping and point dropping to generate positive and negative views for contrastive learning.
• Integrates spatial context by constructing local neighborhood graphs and using them as part of the positive sample formation.
• Employs a Siamese network architecture with shared weights to encode point cloud features and compute contrastive loss.
• Leverages pre-trained features to initialize downstream models for fine-tuning with minimal labeled data.
• Enables active labeling by clustering pre-trained point features to prioritize informative, diverse points for annotation.

Experimental results

Research questions

• RQ1 Can integrating spatial context into contrastive learning improve 3D representation learning for data-efficient scene understanding?
• RQ2 How little labeled data is needed to achieve competitive performance in 3D instance segmentation and semantic segmentation?
• RQ3 Can self-supervised pre-training reduce the need for exhaustive human annotation in 3D point cloud datasets?
• RQ4 Does the proposed method enable effective active learning strategies in cluttered 3D scenes?
• RQ5 How does the performance of the proposed method compare to state-of-the-art approaches like PointContrast under data-scarce settings?

Key findings

• On ScanNet, the method achieves 89% of the baseline mAP@0.5 performance for instance segmentation using only 0.1% of the point labels.
• For semantic segmentation, the method attains 96% of the full-annotation performance with the same 0.1% label budget.
• In the Limited Annotations (LA) setting, the method outperforms PointContrast by 5.4 points in mAP@0.5 on instance segmentation with 200 labeled points per scene.
• On S3DIS, the method achieves 72.2% mIoU in semantic segmentation, surpassing the supervised pre-training result reported in PointContrast.
• In the Limited Scene Reconstructions (LR) setting, fine-tuning with the proposed pre-trained weights improves mAP@0.5 by 16.3 points when training with only 20% of the scenes.
• Active labeling guided by pre-trained features covers more distinct objects in cluttered scenes, demonstrating improved sampling efficiency.