[Paper Review] CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory

Nur Muhammad Mahi Shafiullah, Chris Paxton+3 more|arXiv (Cornell University)|Oct 11, 2022

Domain Adaptation and Few-Shot Learning11 citations

TL;DR

CLIP-Fields introduces a weakly supervised, implicit 3D scene representation that learns semantic embeddings from spatial locations using only web-pretrained vision-language models like CLIP and Detic, enabling zero-shot semantic segmentation, instance identification, and semantic navigation in real-world environments with minimal human annotation. The method achieves strong few-shot performance on HM3D and enables real-robot navigation via natural language queries.

ABSTRACT

We propose CLIP-Fields, an implicit scene model that can be used for a variety of tasks, such as segmentation, instance identification, semantic search over space, and view localization. CLIP-Fields learns a mapping from spatial locations to semantic embedding vectors. Importantly, we show that this mapping can be trained with supervision coming only from web-image and web-text trained models such as CLIP, Detic, and Sentence-BERT; and thus uses no direct human supervision. When compared to baselines like Mask-RCNN, our method outperforms on few-shot instance identification or semantic segmentation on the HM3D dataset with only a fraction of the examples. Finally, we show that using CLIP-Fields as a scene memory, robots can perform semantic navigation in real-world environments. Our code and demonstration videos are available here: https://mahis.life/clip-fields

Motivation & Objective

• To develop a scalable, open-vocabulary 3D spatial-semantic memory for robots that avoids reliance on human-annotated datasets.
• To enable semantic understanding and navigation in 3D environments using only web-scale, pretrained vision-language models as supervision.
• To demonstrate that implicit neural fields can be trained with contrastive loss using features from CLIP and Detic, enabling zero-shot or few-shot scene understanding.
• To enable real-world robotic navigation using natural language queries by querying a learned 3D semantic embedding field.
• To show that performance correlates with the quality of the base vision-language models used during pre-training.

Proposed method

• CLIP-Fields defines a neural implicit function $ g: \mathbb{R}^3 \to \mathbb{R}^d $ that maps 3D spatial coordinates to semantic embedding vectors.
• The model is trained using a contrastive loss that encourages similar features for spatial locations corresponding to the same object or scene region across multiple RGB-D views.
• Features are extracted from pretrained models: CLIP for vision-language alignment, Detic for object detection, and Sentence-BERT for text queries.
• The method leverages multi-view RGB-D data and estimated camera poses to supervise the 3D embedding field without explicit human annotations.
• During inference, queries are encoded into embedding vectors and matched to the nearest point in the CLIP-Fields representation via cosine similarity.
• The robot navigates to the spatial location with the highest alignment between the query embedding and the field’s stored features.

Experimental results

Research questions

• RQ1 Can a 3D scene representation be learned with minimal human supervision by leveraging pre-trained vision-language models?
• RQ2 Can CLIP-Fields enable few-shot instance identification and semantic segmentation without fine-tuning on annotated data?
• RQ3 Can a robot perform semantic navigation using only natural language queries and a weakly supervised 3D embedding field?
• RQ4 How does the performance of CLIP-Fields depend on the quality of the underlying vision-language models used for supervision?
• RQ5 Can CLIP-Fields support open-vocabulary queries, including visual and semantic descriptions, in real-world environments?

Key findings

• CLIP-Fields outperforms Mask-RCNN on few-shot instance identification and semantic segmentation tasks in the HM3D benchmark with only a fraction of the training data.
• The method enables real-world robotic navigation to objects based on natural language queries, with success observed in both kitchen and library environments.
• Performance strongly correlates with the accuracy of the base vision models used during data preparation; misclassified objects in training lead to navigation failures.
• The system successfully retrieves objects under literal queries even with minor spelling variations, indicating robustness to lexical noise.
• Semantic queries sometimes confuse semantically similar concepts (e.g., dishwasher for "wash my hand" or "wash my dishes"), indicating limitations in disambiguation.
• Visual queries can be misled by semantic similarity (e.g., retrieving a white bowl for "red fruit bowl"), but precision improves with more specific descriptions like "red plastic bowl".