[Paper Review] PLA: Language-Driven Open-Vocabulary 3D Scene Understanding

Runyu Ding, Jihan Yang+4 more|arXiv (Cornell University)|Nov 29, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes PLA, a language-driven framework for open-vocabulary 3D scene understanding that leverages vision-language foundation models to generate semantic-rich captions from multi-view images of 3D scenes. By constructing hierarchical point-caption pairs and using contrastive learning, PLA enables zero-shot localization and recognition of unseen categories, achieving 25.8%–44.7% higher hIoU and 14.5%–50.4% higher hAP50 than baselines on ScanNet and S3IDS datasets.

ABSTRACT

Open-vocabulary scene understanding aims to localize and recognize unseen categories beyond the annotated label space. The recent breakthrough of 2D open-vocabulary perception is largely driven by Internet-scale paired image-text data with rich vocabulary concepts. However, this success cannot be directly transferred to 3D scenarios due to the inaccessibility of large-scale 3D-text pairs. To this end, we propose to distill knowledge encoded in pre-trained vision-language (VL) foundation models through captioning multi-view images from 3D, which allows explicitly associating 3D and semantic-rich captions. Further, to foster coarse-to-fine visual-semantic representation learning from captions, we design hierarchical 3D-caption pairs, leveraging geometric constraints between 3D scenes and multi-view images. Finally, by employing contrastive learning, the model learns language-aware embeddings that connect 3D and text for open-vocabulary tasks. Our method not only remarkably outperforms baseline methods by 25.8% $\sim$ 44.7% hIoU and 14.5% $\sim$ 50.4% hAP$_{50}$ in open-vocabulary semantic and instance segmentation, but also shows robust transferability on challenging zero-shot domain transfer tasks. See the project website at https://dingry.github.io/projects/PLA.

Motivation & Objective

• Address the challenge of open-vocabulary 3D scene understanding, where models must recognize unseen semantic categories beyond the training label space.
• Overcome the lack of large-scale 3D-text pairs, which hinders direct transfer of 2D vision-language pre-training to 3D.
• Enable zero-shot generalization to novel categories, including synonyms, abstract concepts, and unannotated objects, in real-world 3D scenes.
• Develop a generic, scalable framework that leverages pre-trained vision-language models to inject rich semantic knowledge into 3D understanding models.
• Achieve robust performance on both in-domain and out-of-domain zero-shot transfer benchmarks without task-specific architectural modifications.

Proposed method

• Use pre-trained vision-language foundation models (e.g., GPT-ViT2, OFA) to generate semantic-rich captions from multi-view images projected from 3D point clouds.
• Construct hierarchical point-caption pairs—scene-level, view-level, and entity-level—using geometric constraints between 3D scenes and their 2D projections.
• Leverage contrastive learning to align 3D point features with corresponding caption embeddings, learning language-aware visual representations.
• Employ a vision-language pre-trained text encoder (e.g., CLIP) to extract more robust textual embeddings that capture visual semantics.
• Utilize neural rendering or existing 3D data pipelines to obtain multi-view RGB images aligned with 3D point clouds for captioning.
• Train the 3D model end-to-end using contrastive loss on the hierarchical point-caption pairs to enable zero-shot generalization.

Experimental results

Research questions

• RQ1 Can pre-trained vision-language models be effectively leveraged to generate semantic-rich captions for 3D scenes without direct 3D-text supervision?
• RQ2 How can hierarchical point-caption pairs, informed by 3D geometry, improve coarse-to-fine visual-semantic representation learning in 3D?
• RQ3 To what extent can contrastive learning on 3D-language pairs enable zero-shot recognition of unseen categories in 3D scene understanding?
• RQ4 Can the proposed framework generalize to out-of-domain datasets with both distribution and category shifts?
• RQ5 How does the quality of foundation model-generated captions compare to human-annotated or curated captions in terms of downstream 3D understanding performance?

Key findings

• PLA achieves 25.8% to 44.7% higher hIoU on open-vocabulary semantic segmentation compared to baseline methods on the ScanNet and S3IDS datasets.
• On instance segmentation, PLA improves hAP50 by 14.5% to 50.4% over baselines, demonstrating strong generalization to novel instances.
• The model generalizes effectively to zero-shot domain transfer, performing well on S3IDS when trained on ScanNet, even with data distribution and category shifts.
• Using CLIP as the text encoder yields over 7% higher mIoU_N on novel categories than BERT or GPT2, confirming the benefit of vision-aware text encodings.
• Foundation models like OFA outperform GPT-ViT2 in captioning, boosting hIoU by up to 0.3 points, showing scalability with better captioning models.
• The model successfully recognizes synonymical classes (e.g., 'couch' for 'sofa'), abstract concepts (e.g., 'bathroom', 'kitchen'), and unannotated objects (e.g., 'monitor', 'blackboard') with high-quality masks.