[Paper Review] Self-Supervised Pretraining of 3D Features on any Point-Cloud

Zaiwei Zhang, Rohit Girdhar+2 more|arXiv (Cornell University)|Jan 7, 2021

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes DepthContrast, a self-supervised pretraining method for 3D point-cloud and voxel-based models using unlabeled single- or multi-view depth maps without requiring 3D registration or correspondences. By jointly training on multiple input formats (points and voxels) with contrastive learning, it achieves state-of-the-art performance on 9 3D recognition benchmarks, including 69.0% mAP on ScanNet detection and 63.5% on SUNRGBD, outperforming supervised pretraining and showing strong label efficiency.

ABSTRACT

Pretraining on large labeled datasets is a prerequisite to achieve good performance in many computer vision tasks like 2D object recognition, video classification etc. However, pretraining is not widely used for 3D recognition tasks where state-of-the-art methods train models from scratch. A primary reason is the lack of large annotated datasets because 3D data is both difficult to acquire and time consuming to label. We present a simple self-supervised pertaining method that can work with any 3D data - single or multiview, indoor or outdoor, acquired by varied sensors, without 3D registration. We pretrain standard point cloud and voxel based model architectures, and show that joint pretraining further improves performance. We evaluate our models on 9 benchmarks for object detection, semantic segmentation, and object classification, where they achieve state-of-the-art results and can outperform supervised pretraining. We set a new state-of-the-art for object detection on ScanNet (69.0% mAP) and SUNRGBD (63.5% mAP). Our pretrained models are label efficient and improve performance for classes with few examples.

Motivation & Objective

• To address the lack of large-scale annotated 3D datasets by enabling self-supervised pretraining on easily acquired single-view depth maps.
• To develop a method that works across diverse 3D data types—indoor/outdoor, single/multi-view, various sensors—without requiring 3D registration or correspondences.
• To improve 3D representation learning by jointly pretraining on multiple input formats (e.g., point clouds and voxels) using contrastive learning.
• To demonstrate that self-supervised pretraining can outperform supervised pretraining and significantly improve few-shot learning performance on 3D recognition tasks.

Proposed method

• The method uses a contrastive learning framework where each depth map is treated as a unique instance, and the model learns to discriminate between different augmented views of the same depth map.
• It employs format-specific encoders (e.g., PointNet for point clouds, SpConv-UNet for voxels) to extract spatial features from the same input represented in different formats.
• Global features are derived via global max pooling and a two-layer MLP, which are used to compute contrastive loss between positive and negative pairs.
• The model is jointly trained using both within-format and across-format contrastive losses to encourage consistency across different input representations.
• Data augmentation is applied to depth maps before conversion into point clouds or voxels, with specific strategies like random cuboid cropping and color jittering.
• The approach is agnostic to sensor type and scene type, enabling pretraining on diverse data such as indoor RGB-D scans, outdoor LiDAR, and single-view depth videos.

Experimental results

Research questions

• RQ1 Can self-supervised pretraining on unlabeled single-view depth maps improve 3D representation learning without requiring 3D registration or multi-view correspondences?
• RQ2 Does joint pretraining on multiple input formats (e.g., points and voxels) lead to better generalization than pretraining on a single format?
• RQ3 Can the proposed method generalize across different 3D data types, including indoor and outdoor scenes, and various sensors such as RGB-D and LiDAR?
• RQ4 How does the performance of the self-supervised pretraining compare to supervised pretraining and training from scratch, especially in low-data regimes?

Key findings

• DepthContrast achieves a new state-of-the-art mAP of 69.0% on the ScanNet 3D object detection benchmark, surpassing prior methods.
• It sets a new SOTA of 63.5% mAP on the SUNRGBD detection benchmark, outperforming previous methods and even supervised pretraining.
• On the KITTI dataset, the model achieves a 20% performance gain when fine-tuned on only 5% of the labeled data, demonstrating strong label efficiency.
• Pretraining on single-view depth maps (e.g., ScanNet-vid or Redwood-vid) yields performance comparable to pretraining on multi-view registered 3D scenes (e.g., ScanNet), showing robustness to input preprocessing.
• Joint pretraining on both point-cloud and voxel representations leads to better downstream performance than pretraining on a single format, highlighting the importance of multi-format learning.
• The method generalizes to outdoor LiDAR data, as shown by strong performance on the KITTI cyclist detection task using pretraining on the Waymo dataset.