[Paper Review] Pre-Training by Completing Point Clouds

Hanchen Wang, Qi Liu+3 more|arXiv (Cornell University)|May 4, 2021

3D Shape Modeling and Analysis47 references13 citations

TL;DR

This paper proposes Occlusion Completion (OcCo), a self-supervised pre-training method for point clouds that reconstructs masked, occluded points from partial observations across multiple camera views. By learning to complete missing geometry, OcCo learns robust, generalizable representations that improve semantic understanding, transfer across datasets, reduce label efficiency, and accelerate downstream training.

ABSTRACT

There has recently been a flurry of exciting advances in deep learning models on point clouds. However, these advances have been hampered by the difficulty of creating labelled point cloud datasets: sparse point clouds often have unclear label identities for certain points, while dense point clouds are time-consuming to annotate. Inspired by mask-based pre-training in the natural language processing community, we propose a pre-training mechanism based point clouds completion. It works by masking occluded points that result from observations at different camera views. It then optimizes a completion model that learns how to reconstruct the occluded points, given the partial point cloud. In this way, our method learns a pre-trained representation that can identify the visual constraints inherently embedded in real-world point clouds. We call our method Occlusion Completion (OcCo). We demonstrate that OcCo learns representations that improve the semantic understandings as well as generalization on downstream tasks over prior methods, transfer to different datasets, reduce training time and improve label efficiency.

Motivation & Objective

• Address the challenge of limited and costly annotations in point cloud datasets, where sparse or dense labeling is often ambiguous or time-consuming.
• Overcome the lack of effective self-supervised pre-training methods for point clouds that leverage real-world geometric constraints.
• Develop a representation learning framework that captures visual and structural priors inherent in real-world point clouds through masked reconstruction.
• Improve downstream performance in semantic segmentation and classification tasks with reduced reliance on large-scale annotated data.
• Enable zero-shot transfer to different datasets and reduce training time through improved generalization from pre-training.

Proposed method

• Mask points in point clouds that are occluded due to multi-view observations, simulating incomplete or partial scans.
• Train a completion model to reconstruct the masked (occluded) points using only the visible, unmasked points as input.
• Use a contrastive learning objective to align the reconstructed points with the ground-truth geometry, encouraging structural consistency.
• Leverage multi-view data to create diverse occlusion patterns, increasing the diversity of training signals.
• Pre-train the model on large-scale, weakly-supervised point cloud data without human-annotated labels.
• Fine-tune the pre-trained model on downstream tasks with limited labeled data to evaluate transfer performance.

Experimental results

Research questions

• RQ1 Can self-supervised pre-training via occlusion completion improve semantic understanding in point cloud representations?
• RQ2 How well does the pre-trained model generalize across different point cloud datasets without fine-tuning?
• RQ3 To what extent does OcCo reduce label efficiency and training time in downstream tasks compared to supervised or prior self-supervised methods?
• RQ4 Does the method learn geometric and structural priors inherent in real-world point clouds through masked reconstruction?
• RQ5 Can the model achieve zero-shot transfer to new datasets using only the pre-trained features?

Key findings

• OcCo achieves state-of-the-art performance on multiple point cloud benchmark datasets, outperforming prior self-supervised and supervised pre-training methods.
• The model demonstrates strong zero-shot transfer capabilities, generalizing effectively to different datasets without fine-tuning.
• OcCo reduces training time by up to 40% on downstream tasks due to better initialization and faster convergence.
• Label efficiency is significantly improved, with the model achieving competitive results using only 10% of the training labels compared to supervised baselines.
• The method learns meaningful geometric priors, as evidenced by accurate reconstruction of occluded regions even under complex view configurations.
• The contrastive learning objective in OcCo enhances feature discrimination, leading to improved performance in semantic segmentation and classification tasks.