[Paper Review] EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Grounding

Yanmin Wu, Xinhua Cheng+3 more|arXiv (Cornell University)|Sep 29, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes EDA, a novel 3D visual grounding framework that explicitly decouples text into semantic components—such as attributes, relations, and pronouns—and enforces dense alignment between these components and 3D point cloud objects via position and semantic alignment losses. The method achieves state-of-the-art performance on ScanRefer and SR3D/NR3D, with absolute leadership on a new challenging task: grounding without object names, demonstrating robust, fine-grained visual-linguistic understanding beyond reliance on object categories.

ABSTRACT

3D visual grounding aims to find the object within point clouds mentioned by free-form natural language descriptions with rich semantic cues. However, existing methods either extract the sentence-level features coupling all words or focus more on object names, which would lose the word-level information or neglect other attributes. To alleviate these issues, we present EDA that Explicitly Decouples the textual attributes in a sentence and conducts Dense Alignment between such fine-grained language and point cloud objects. Specifically, we first propose a text decoupling module to produce textual features for every semantic component. Then, we design two losses to supervise the dense matching between two modalities: position alignment loss and semantic alignment loss. On top of that, we further introduce a new visual grounding task, locating objects without object names, which can thoroughly evaluate the model's dense alignment capacity. Through experiments, we achieve state-of-the-art performance on two widely-adopted 3D visual grounding datasets, ScanRefer and SR3D/NR3D, and obtain absolute leadership on our newly-proposed task. The source code is available at https://github.com/yanmin-wu/EDA.

Motivation & Objective

• To address the imbalance and ambiguity in existing 3D visual grounding methods that implicitly couple all words into a single sentence-level feature.
• To reduce inductive bias toward object names by decoupling linguistic components such as attributes, relations, and pronouns.
• To enable fine-grained, component-level alignment between text and 3D point cloud objects through explicit supervision.
• To propose a new benchmark task—grounding without object names—to rigorously evaluate model robustness and generalization.
• To demonstrate that models can accurately localize objects using only attributes and spatial relationships, without relying on category-level cues.

Proposed method

• A text decoupling module parses input sentences into distinct semantic components: main object, attributes, relations, pronouns, and auxiliary objects.
• Two supervision losses are introduced: position alignment loss to align predicted bounding boxes with ground-truth, and semantic alignment loss to match textual and visual features at the component level.
• The model performs dense matching between each decoupled text component and candidate 3D objects, selecting the object with highest overall similarity.
• The method is trained end-to-end using cross-attention mechanisms and contrastive learning to enhance feature alignment.
• A new evaluation protocol is introduced where object names are replaced with 'object' to create a challenging zero-shot-like setting.
• The model is evaluated on both standard 3D VG benchmarks and the new VG-w/o-ON task without retraining.

Experimental results

Research questions

• RQ1 Can explicit decoupling of linguistic components improve 3D visual grounding performance by reducing inductive bias toward object names?
• RQ2 Does dense, component-level alignment between text and 3D objects lead to better localization accuracy than sentence-level feature fusion?
• RQ3 Can models achieve robust grounding when object names are absent, relying only on attributes and spatial relationships?
• RQ4 How do different text components (e.g., attributes, relations) contribute to grounding performance in the absence of object names?
• RQ5 To what extent does the proposed method generalize beyond standard benchmarks to more challenging, realistic language descriptions?

Key findings

• EDA achieves state-of-the-art performance on ScanRefer and SR3D/NR3D, with 54.6% and 42.3% accuracy at 0.25 IoU and 0.5 IoU, respectively.
• On the new VG-w/o-ON task, EDA achieves 26.5% and 21.2% accuracy at 0.25 and 0.5 IoU, exceeding all baselines by over 10 percentage points.
• The performance gain from adding text components is more significant in the VG-w/o-ON setting, especially for spatial relationships, which provide strong localization cues.
• Ablation studies confirm that all decoupled components contribute to performance, with the relationship component being most impactful in the no-name setting.
• Qualitative results show EDA can correctly localize objects without knowing their names, while baselines like BUTD-DETR fail catastrophically in this setting.
• The model generalizes well to complex descriptions involving both attributes and relations, outperforming baselines that struggle with ambiguity in such cases.