[Paper Review] <b>D</b>ataset for <b>O</b>pen <b>V</b>ocabulary <b>E</b>ntity <b>G</b>rounding (DOVE-G)

Haonan Chang, Kowndinya Boyalakuntla+9 more|arXiv (Cornell University)|Jan 1, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces Open-Vocabulary 3D Scene Graph (OVSG), a novel framework for context-aware entity grounding using free-form text queries in 3D environments. By combining open-vocabulary detection with 3D scene graphs that encode spatial relationships and attributes, OVSG achieves superior performance over prior methods, especially in handling unseen categories and complex queries, as validated on ScanNet, ICL-NUIM, and the new DOVE-G dataset with 4,000 queries across eight scenarios.

ABSTRACT

<b>DOVE-G</b>To accommodate the richness of open-vocabulary queries, we introduced a custom dataset—DOVE-G (Dataset for Open-Vocabulary Entity Grounding). This dataset has 8 scenes namely kitchen, kitchenette, room1, room2, room3, bathroom, computer lab, and hallway. This dataset is created to facilitate users to query for objects within a scene using natural language. For each scene within DOVE-G, we manually labeled the ground truth and created 50 natural language queries (<i>L</i>_<em>q</em> ). To augment this query set, we harnessed LLMs to generate four additional sets of natural language queries. This approach yielded a total of 250 queries for each scene, and cumulatively, we have 4000 queries to evaluate OVSG’s performance. With this setup, we set out to assess how our OVSG framework performs with open-vocabulary queries, one of our key research questions, providing a critical testbed for its effectiveness in handling diverse natural language expressions.

Motivation & Objective

• Address the challenge of grounding entities in 3D scenes when semantic categories, attributes, or relationships are unseen or not predefined.
• Enable context-aware grounding of entities using free-form natural language queries, such as 'pick up a cup on a kitchen table' or 'navigate to a sofa where someone is sitting'.
• Develop a scalable and practical system for real-world robot navigation and manipulation by supporting open-vocabulary queries beyond fixed vocabularies.
• Create a new benchmark dataset, DOVE-G, with 4,000 diverse, context-rich queries across eight real-world indoor scenes to support evaluation of open-vocabulary grounding.
• Demonstrate the real-world applicability of the framework through robot navigation and manipulation experiments in complex, dynamic environments.

Proposed method

• Construct an Open-Vocabulary 3D Scene Graph (OVSG) that represents entities (objects, agents, regions) as nodes and spatial/semantic relationships as edges, with attributes like color, material, and affordance.
• Integrate open-vocabulary vision-language models (e.g., CLIP-based Detic and OVIR-3D) to detect and embed 3D object instances with zero-shot generalization to unseen categories.
• Use 3D global data association to fuse 2D detection results (from OVIR-3D) into a 3D scene, assigning unique features and spatial context to each instance.
• Represent queries as natural language inputs and encode them using a vision-language encoder to align with scene graph nodes and edges.
• Perform grounding by computing cross-attention between the query embedding and the scene graph, selecting the most relevant entity based on similarity scores.
• Apply a multi-scale inference strategy (OVSG-J, OVSG-S, OVSG-L) to balance speed and accuracy, with OVSG-L using full feature fusion and context reasoning.

Figure 1: This is an illustration of the proposed pipeline. The system inputs are the positional input $P_{u}$ , user input $L_{u}$ , RGBD Scan $I$ , and a query language $L_{q}$ . The top section depicts the construction of $G_{s}$ . Both $P_{u}$ and $L_{u}$ are directly fed into $G_{s}$ . The RGBD

Experimental results

Research questions

• RQ1 Can a 3D scene graph representation support context-aware entity grounding for free-form, open-vocabulary queries that include unseen categories and complex spatial relationships?
• RQ2 How does the integration of open-vocabulary detection with 3D scene graphs improve grounding accuracy compared to state-of-the-art semantic-based localization methods?
• RQ3 To what extent can the proposed framework generalize to unseen object categories and novel relationships not present in the training data?
• RQ4 How effective is the system in real-world robot navigation and manipulation tasks requiring precise grounding of ambiguous or context-dependent queries?
• RQ5 What is the performance gain of OVSG over baselines like ConceptFusion and OVIR-3D across diverse 3D environments and IoU thresholds?

Key findings

• OVSG-L achieved a 3D grounding success rate of 100% at IoU > 0.15 on the office_room_traj0_frei_png scene (29 queries), significantly outperforming ConceptFusion (0%) and OVIR-3D (65.52%).
• On the living_room_traj3_frei_png scene (17 queries), OVSG-L achieved 82.36% success rate at IoU > 0.15 and 64.71% at IoU > 0.25, while ConceptFusion failed entirely (0% success rate).
• OVSG-L achieved 100% success rate at IoU > 0.15 across all scenes in the office_room_traj0_frei_png and office_room_traj1_frei_png, demonstrating robustness in complex indoor environments.
• The DOVE-G dataset contains 4,000 queries across eight unique indoor scenes, with 50 diverse, context-rich queries per scene, including complex spatial and relational descriptions.
• OVSG-L outperformed OVIR-3D and ConceptFusion across all IoU thresholds (0.15, 0.25, 0.5, 0.75) on both 'Object-only' and 'Whole Query' categories in ICL-NUIM, especially in handling full-sentence queries.
• Qualitative results show OVSG-L correctly grounds entities in complex, ambiguous queries (e.g., 'a cup on a table near a person'), while ConceptFusion often fails due to lack of context modeling.

Figure 2: Architecture of spatial-language encoder and predictor. The blue block is the spatial pose encoder, and the yellow block is the spatial relationship predictor.