[Paper Review] Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training

Xiaoyang Wu, Zhuotao Tian+5 more|arXiv (Cornell University)|Aug 18, 2023

3D Shape Modeling and Analysis4 citations

TL;DR

This paper proposes Point Prompt Training (PPT), a novel framework for large-scale 3D representation learning that enables multi-dataset synergistic training by introducing Prompt-driven Normalization and Language-guided Categorical Alignment to mitigate negative transfer. PPT achieves state-of-the-art performance across diverse indoor and outdoor 3D benchmarks using a single, shared-weight model, significantly improving generalization and representation quality over single-dataset or naive multi-dataset training approaches.

ABSTRACT

The rapid advancement of deep learning models often attributes to their ability to leverage massive training data. In contrast, such privilege has not yet fully benefited 3D deep learning, mainly due to the limited availability of large-scale 3D datasets. Merging multiple available data sources and letting them collaboratively train a single model is a potential solution. However, due to the large domain gap between 3D point cloud datasets, such mixed supervision could adversely affect the model's performance and lead to degenerated performance (i.e., negative transfer) compared to single-dataset training. In view of this challenge, we introduce Point Prompt Training (PPT), a novel framework for multi-dataset synergistic learning in the context of 3D representation learning that supports multiple pre-training paradigms. Based on this framework, we propose Prompt-driven Normalization, which adapts the model to different datasets with domain-specific prompts and Language-guided Categorical Alignment that decently unifies the multiple-dataset label spaces by leveraging the relationship between label text. Extensive experiments verify that PPT can overcome the negative transfer associated with synergistic learning and produce generalizable representations. Notably, it achieves state-of-the-art performance on each dataset using a single weight-shared model with supervised multi-dataset training. Moreover, when served as a pre-training framework, it outperforms other pre-training approaches regarding representation quality and attains remarkable state-of-the-art performance across over ten diverse downstream tasks spanning both indoor and outdoor 3D scenarios.

Motivation & Objective

• To address the challenge of negative transfer in multi-dataset 3D representation learning due to large domain gaps between point cloud datasets.
• To enable effective synergistic training across multiple 3D datasets by unifying their label spaces and adapting models to domain-specific distributions.
• To develop a universal pre-training framework compatible with both supervised and self-supervised paradigms for 3D point cloud learning.
• To improve generalization and representation quality by leveraging textual relationships between class labels and domain-specific prompt adaptation.

Proposed method

• Introduces Point Prompt Training (PPT), a unified framework for multi-dataset 3D representation learning that supports both supervised and unsupervised pre-training.
• Proposes Prompt-driven Normalization, which applies domain-specific prompts to adapt the model’s normalization layers to different datasets, reducing domain shift impact.
• Introduces Language-guided Categorical Alignment (LCA), which aligns multiple dataset label spaces using textual embeddings of class names to unify supervision signals.
• Employs a shared-weight model trained on multiple datasets, with prompt adapters and LCA enabling domain generalization without performance degradation.
• Utilizes contrastive learning and cross-attention mechanisms to align point features with category-text embeddings in a unified latent space.
• Supports both direct evaluation on included datasets and transfer learning to downstream tasks via fine-tuning.

Experimental results

Research questions

• RQ1 Can multi-dataset training in 3D representation learning overcome negative transfer caused by large domain gaps between datasets?
• RQ2 How can domain-specific distribution shifts in 3D point clouds be effectively mitigated during joint training?
• RQ3 Can a unified label space be constructed across diverse 3D datasets using only textual class descriptions?
• RQ4 Does a single, shared-weight model trained on multiple datasets outperform single-dataset baselines in terms of generalization and downstream performance?
• RQ5 Can the proposed framework serve as a superior pre-training strategy across diverse indoor and outdoor 3D scenarios?

Key findings

• PPT achieves state-of-the-art performance on ScanNet and S3DIS using a single shared-weight model trained on multiple indoor datasets, with 75.4% mIoU on ScanNet validation and 77.5% on test.
• On outdoor benchmarks, PPT achieves SOTA results on SemanticKITTI (88.34% mIoU), nuScenes (90.84% mIoU), and Waymo (94.56% mIoU), demonstrating strong generalization across domains.
• PPT outperforms naive joint training, which suffers from negative transfer, by mitigating performance degradation through domain prompt adaptation and label space alignment.
• The ablation study confirms that shared domain prompts improve generalization, while block-wise independent prompts lead to overfitting and performance drop.
• Language-guided Categorical Alignment (LCA) acts as an effective prediction head in single-dataset training, improving mIoU from 73.4 to 74.2 on SpUNet-S.
• Scaling up the backbone with PPT reduces overfitting and improves performance, with SpUNet-L achieving 75.8% mIoU on S3DIS, outperforming smaller models without PPT.