[論文レビュー] Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling

Xumin Yu, Lulu Tang+4 more|arXiv (Cornell University)|Nov 29, 2021

3D Shape Modeling and Analysis参考文献 52被引用数 49

ひとこと要約

Point-BERTは、Masked Point ModelingタスクとdVAEを介して学習される離散点トークンボキャブラリを用いて3D点群Transformersを事前学習し、ModelNet40とScanObjectNNで優れた結果を達成し、新しいタスクへの良好な転移を可能にします。

ABSTRACT

We present Point-BERT, a new paradigm for learning Transformers to generalize the concept of BERT to 3D point cloud. Inspired by BERT, we devise a Masked Point Modeling (MPM) task to pre-train point cloud Transformers. Specifically, we first divide a point cloud into several local point patches, and a point cloud Tokenizer with a discrete Variational AutoEncoder (dVAE) is designed to generate discrete point tokens containing meaningful local information. Then, we randomly mask out some patches of input point clouds and feed them into the backbone Transformers. The pre-training objective is to recover the original point tokens at the masked locations under the supervision of point tokens obtained by the Tokenizer. Extensive experiments demonstrate that the proposed BERT-style pre-training strategy significantly improves the performance of standard point cloud Transformers. Equipped with our pre-training strategy, we show that a pure Transformer architecture attains 93.8% accuracy on ModelNet40 and 83.1% accuracy on the hardest setting of ScanObjectNN, surpassing carefully designed point cloud models with much fewer hand-made designs. We also demonstrate that the representations learned by Point-BERT transfer well to new tasks and domains, where our models largely advance the state-of-the-art of few-shot point cloud classification task. The code and pre-trained models are available at https://github.com/lulutang0608/Point-BERT

研究の動機と目的

• 最小限の帰納バイアスで3D点群へBERTスタイルの事前学習を拡張する動機付け。
• 局所的な点パッチを離散トークンへ変換するトークナイゼーション機構を開発する。
• マスク付き点モデリングの事前学習目的を提案し、マスクされたトークンを復元する。
• 高レベルの意味論を捉える補助的なコントラスト学習目的で表現を強化する。
• 点群タスクに対する強力な転移、少数ショット、および実世界の性能向上を実証する。）
• method (著者が定義した各要素を日本語訳):
• 参考行の配列を以下に翻訳します。
• - FPSとkNNグルーピングによって3D点群を局所パッチ（サブクラウド）に分割する。
• - サブクラウドをミニPointNetで埋め込みへ射影し、パッチ埋め込みの系列を形成する。
• - 埋め込みを離散点トークンへ変換するトークナイザーを離散VAE (dVAE)で学習する。
• - dVAEの監視信号を用いてパッチをマスクし、トークンを再構成することでMasked Point ModelingでTransformerバックボーンを事前学習する。
• - ブロック単位のマスキング戦略を適用し、事前学習中に学習可能なマスクトークンを使用する。
• - 高レベルの意味表現を促進するため、Point Patch Mixingを用いたMoCoベースのコントラスト損失。

提案手法

• Partition a 3D point cloud into local patches (sub-clouds) via FPS and kNN grouping.
• Project sub-clouds into embeddings with a mini-PointNet and form a sequence of patch embeddings.
• Learn a Tokenizer with a discrete VAE (dVAE) to convert embeddings into discrete point tokens.
• Pre-train a Transformer backbone with Masked Point Modeling by masking patches and reconstructing tokens using the dVAE supervision.
• Apply a block-wise masking strategy and use a learnable mask token during pre-training.
• Incorporate a MoCo-based contrastive loss with Point Patch Mixing to encourage high-level semantic representations.

実験結果

リサーチクエスチョン

• RQ1 Can a BERT-style pre-training objective be effectively applied to 3D point clouds using discrete tokens?
• RQ2 Do discrete point tokens learned via dVAE capture meaningful local geometric patterns for representation learning?
• RQ3 Does Masked Point Modeling, aided by contrastive learning and patch mixing, improve downstream 3D tasks compared to training from scratch?
• RQ4 How well do Point-BERT representations transfer to real-world datasets and few-shot scenarios?

主な発見

• Point-BERT achieves 93.8% accuracy on ModelNet40 with more input points, outperforming several hand-crafted and transformer-based baselines.
• On the challenging ScanObjectNN setting, Point-BERT reaches 83.1% accuracy, surpassing prior models with fewer hand-designed biases.
• Pre-training with Point-BERT consistently improves Transformer performance over training from scratch and scales with input density (e.g., 93.4% with 4096 pts, 93.8% with 8192 pts).
• Point-BERT representations transfer well to new tasks and domains, advancing state-of-the-art in few-shot point cloud classification.
• Ablation studies show that the combination of MPM, Point Patch Mixing, and MoCo yields the strongest performance gains.