[Paper Review] LanguageRefer: Spatial-Language Model for 3D Visual Grounding

Junha Roh, Karthik Desingh+2 more|arXiv (Cornell University)|Jul 7, 2021

Multimodal Machine Learning Applications43 references22 citations

TL;DR

LanguageRefer proposes a transformer-based spatial-language model for 3D visual grounding that fuses spatial embeddings from 3D bounding boxes with fine-tuned DistilBERT embeddings to identify referred objects in noisy 3D scenes. It achieves state-of-the-art performance on ReferIt3D by decoupling spatial reasoning from perception, with a 48.2% accuracy on Nr3D and a 12.7% improvement on view-dependent utterances when using corrected viewpoint annotations.

ABSTRACT

For robots to understand human instructions and perform meaningful tasks in the near future, it is important to develop learned models that comprehend referential language to identify common objects in real-world 3D scenes. In this paper, we introduce a spatial-language model for a 3D visual grounding problem. Specifically, given a reconstructed 3D scene in the form of point clouds with 3D bounding boxes of potential object candidates, and a language utterance referring to a target object in the scene, our model successfully identifies the target object from a set of potential candidates. Specifically, LanguageRefer uses a transformer-based architecture that combines spatial embedding from bounding boxes with fine-tuned language embeddings from DistilBert to predict the target object. We show that it performs competitively on visio-linguistic datasets proposed by ReferIt3D. Further, we analyze its spatial reasoning task performance decoupled from perception noise, the accuracy of view-dependent utterances, and viewpoint annotations for potential robotics applications.

Motivation & Objective

• To develop a modular, end-to-end spatial-language model that improves 3D visual grounding by decoupling spatial reasoning from perception noise.
• To investigate the impact of viewpoint ambiguity in view-dependent utterances on model performance in 3D visual grounding.
• To enhance model robustness and interpretability by introducing corrected viewpoint annotations for view-dependent utterances.
• To evaluate the model’s generalization across different datasets and its potential for robotics applications involving referential language.
• To analyze the contribution of individual loss components and the role of class label quality in model accuracy.

Proposed method

• The model uses a transformer-based architecture that jointly encodes language utterances and 3D spatial information from object bounding boxes.
• Spatial embeddings are derived from 3D bounding box coordinates using positional encoding to represent spatial relationships.
• Language embeddings are obtained from fine-tuned DistilBERT, which processes natural language descriptions of target objects.
• The spatial-language model predicts the target object by attending to the most relevant object candidates based on multimodal embeddings.
• A modular pipeline separates perception (object class labels), spatial embedding, language embedding, and the spatial-language reasoning module for flexibility and interpretability.
• The authors introduce corrected viewpoint annotations for view-dependent utterances by collecting human-validated orientations, enabling consistent scene alignment during training and inference.

Experimental results

Research questions

• RQ1 How does decoupling spatial reasoning from perception noise affect 3D visual grounding performance?
• RQ2 To what extent does viewpoint ambiguity in view-dependent utterances degrade model accuracy, and can it be mitigated?
• RQ3 How effective is the model when using ground-truth class labels versus noisy predictions in real-world 3D scenes?
• RQ4 What is the impact of different loss components (classification, mask, text) on model performance and generalization?
• RQ5 Can the model generalize to different datasets and support robotics applications involving referential language instructions?

Key findings

• LanguageRefer achieved a 48.2% accuracy on the Nr3D benchmark, outperforming state-of-the-art models without additional training data.
• The model showed a 12.7% accuracy improvement on view-dependent utterances when using corrected viewpoint annotations, indicating strong gains from consistent spatial alignment.
• The model achieved 40.6% accuracy on the Sr3D dataset with predicted class labels, demonstrating robustness to synthetic, template-based language.
• Ablation studies revealed that the classification loss improved performance, while the text loss degraded accuracy, leading to the removal of text loss terms in the final model.
• The model trained with corrected orientations generalized well, achieving 50.7% accuracy on the overall test set, with a 5.1% improvement over the baseline without correction.
• The model maintained high performance on view-independent utterances (46.4% accuracy), indicating strong generalization to viewpoint-agnostic references.