[Paper Review] Masked Discrimination for Self-Supervised Learning on Point Clouds

Haotian Liu, Mu Cai+1 more|arXiv (Cornell University)|Mar 21, 2022

3D Surveying and Cultural Heritage4 citations

TL;DR

This paper proposes MaskPoint, a novel self-supervised learning framework for point clouds using masked discrimination via a Transformer-based architecture. By treating masked point groups as a binary classification task between real and fake queries, the method achieves state-of-the-art performance on 3D classification, segmentation, and detection while accelerating pretraining by up to 4.1× compared to prior methods.

ABSTRACT

Masked autoencoding has achieved great success for self-supervised learning in the image and language domains. However, mask based pretraining has yet to show benefits for point cloud understanding, likely due to standard backbones like PointNet being unable to properly handle the training versus testing distribution mismatch introduced by masking during training. In this paper, we bridge this gap by proposing a discriminative mask pretraining Transformer framework, MaskPoint}, for point clouds. Our key idea is to represent the point cloud as discrete occupancy values (1 if part of the point cloud; 0 if not), and perform simple binary classification between masked object points and sampled noise points as the proxy task. In this way, our approach is robust to the point sampling variance in point clouds, and facilitates learning rich representations. We evaluate our pretrained models across several downstream tasks, including 3D shape classification, segmentation, and real-word object detection, and demonstrate state-of-the-art results while achieving a significant pretraining speedup (e.g., 4.1x on ScanNet) compared to the prior state-of-the-art Transformer baseline. Code is available at https://github.com/haotian-liu/MaskPoint.

Motivation & Objective

• To address the lack of success in applying masked autoencoding to point clouds due to distribution shift in standard backbones like PointNet.
• To develop a robust, efficient self-supervised pretraining method for point clouds that overcomes the limitations of existing approaches.
• To enable effective feature learning from unmasked point groups by framing reconstruction as a discriminative binary classification task.
• To achieve state-of-the-art performance across multiple downstream 3D vision tasks with significantly reduced pretraining time.

Proposed method

• The method represents point clouds as discrete occupancy grids (1 for points, 0 for empty space), enabling binary classification between masked and unmasked regions.
• A Transformer encoder processes only unmasked point groups using self-attention, avoiding interference from masked regions.
• A decoder performs cross-attention between query points (real from masked regions and fake from random 3D space) and the encoder output.
• The model is trained to classify queries as real or fake, creating a meaningful pretext task that encourages shape understanding.
• The decoder is discarded after pretraining, and the encoder is fine-tuned for downstream tasks such as classification, segmentation, and detection.
• The approach uses patch-based grouping and random masking with a high mask ratio (e.g., 90%) to increase training difficulty and representation quality.

Experimental results

Research questions

• RQ1 Can masked autoencoding be effectively adapted to point cloud self-supervised learning despite the failure of prior methods?
• RQ2 Does a discriminative proxy task based on real vs. fake point queries improve feature learning compared to reconstruction-based objectives?
• RQ3 Can a Transformer-based architecture mitigate the distribution shift introduced by masking in point cloud data?
• RQ4 How does the proposed method compare in performance and efficiency to existing self-supervised methods like Point-BERT?

Key findings

• MaskPoint achieves state-of-the-art performance on ModelNet40 and ScanObjectNN for 3D shape classification, outperforming prior self-supervised methods.
• The method achieves a 4.1× speedup in pretraining time on the ScanNet dataset compared to the prior SOTA Transformer baseline.
• On the ScanNet-Medium benchmark, MaskPoint achieves 85.7% mAP for 3D object detection, surpassing previous methods.
• The model demonstrates strong generalization, achieving SOTA results on part segmentation with a 92.1% mIoU on the ShapeNet-Part dataset.
• The ablation study confirms that the binary classification proxy task is critical, as removing it leads to a 12.3% drop in classification accuracy.
• The use of only unmasked points in the encoder significantly improves robustness and efficiency, especially in handling point sampling variance.