[Paper Review] Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling

Xumin Yu, Lulu Tang+4 more|arXiv (Cornell University)|Nov 29, 2021

3D Shape Modeling and AnalysisEngineering52 references49 citations

TL;DR

Point-BERT pre-trains 3D point cloud Transformers using a Masked Point Modeling task and a discrete point token vocabulary learned via a dVAE, achieving strong results on ModelNet40 and ScanObjectNN and enabling good transfer to new tasks.

ABSTRACT

We present Point-BERT, a new paradigm for learning Transformers to generalize the concept of BERT to 3D point cloud. Inspired by BERT, we devise a Masked Point Modeling (MPM) task to pre-train point cloud Transformers. Specifically, we first divide a point cloud into several local point patches, and a point cloud Tokenizer with a discrete Variational AutoEncoder (dVAE) is designed to generate discrete point tokens containing meaningful local information. Then, we randomly mask out some patches of input point clouds and feed them into the backbone Transformers. The pre-training objective is to recover the original point tokens at the masked locations under the supervision of point tokens obtained by the Tokenizer. Extensive experiments demonstrate that the proposed BERT-style pre-training strategy significantly improves the performance of standard point cloud Transformers. Equipped with our pre-training strategy, we show that a pure Transformer architecture attains 93.8% accuracy on ModelNet40 and 83.1% accuracy on the hardest setting of ScanObjectNN, surpassing carefully designed point cloud models with much fewer hand-made designs. We also demonstrate that the representations learned by Point-BERT transfer well to new tasks and domains, where our models largely advance the state-of-the-art of few-shot point cloud classification task. The code and pre-trained models are available at https://github.com/lulutang0608/Point-BERT

Motivation & Objective

• Motivate extending BERT-style pre-training to 3D point clouds with minimal inductive bias.
• Develop a tokenization mechanism to convert local point patches into discrete tokens.
• Propose a masked point modeling pre-training objective to recover masked tokens.
• Enhance representations with an auxiliary contrastive objective to capture high-level semantics.
• Demonstrate strong transfer, few-shot, and real-world performance gains for point cloud tasks.

Proposed method

• Partition a 3D point cloud into local patches (sub-clouds) via FPS and kNN grouping.
• Project sub-clouds into embeddings with a mini-PointNet and form a sequence of patch embeddings.
• Learn a Tokenizer with a discrete VAE (dVAE) to convert embeddings into discrete point tokens.
• Pre-train a Transformer backbone with Masked Point Modeling by masking patches and reconstructing tokens using the dVAE supervision.
• Apply a block-wise masking strategy and use a learnable mask token during pre-training.
• Incorporate a MoCo-based contrastive loss with Point Patch Mixing to encourage high-level semantic representations.

Experimental results

Research questions

• RQ1 Can a BERT-style pre-training objective be effectively applied to 3D point clouds using discrete tokens?
• RQ2 Do discrete point tokens learned via dVAE capture meaningful local geometric patterns for representation learning?
• RQ3 Does Masked Point Modeling, aided by contrastive learning and patch mixing, improve downstream 3D tasks compared to training from scratch?
• RQ4 How well do Point-BERT representations transfer to real-world datasets and few-shot scenarios?

Key findings

• Point-BERT achieves 93.8% accuracy on ModelNet40 with more input points, outperforming several hand-crafted and transformer-based baselines.
• On the challenging ScanObjectNN setting, Point-BERT reaches 83.1% accuracy, surpassing prior models with fewer hand-designed biases.
• Pre-training with Point-BERT consistently improves Transformer performance over training from scratch and scales with input density (e.g., 93.4% with 4096 pts, 93.8% with 8192 pts).
• Point-BERT representations transfer well to new tasks and domains, advancing state-of-the-art in few-shot point cloud classification.
• Ablation studies show that the combination of MPM, Point Patch Mixing, and MoCo yields the strongest performance gains.