[Paper Review] Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information

Jialu Li, Hao Tan+1 more|arXiv (Cornell University)|Apr 19, 2021

Multimodal Machine Learning Applications41 references4 citations

TL;DR

This paper proposes a syntax-aware vision-language navigation agent that leverages dependency tree structures via ChildSum Tree-LSTM to improve cross-modal alignment between natural language instructions and visual scenes. By encoding phrase-level syntactic information, the model achieves state-of-the-art performance on the multilingual Room-Across-Room dataset and significantly improves generalization to unseen environments, especially in zero-shot settings.

ABSTRACT

Vision language navigation is the task that requires an agent to navigate through a 3D environment based on natural language instructions. One key challenge in this task is to ground instructions with the current visual information that the agent perceives. Most of the existing work employs soft attention over individual words to locate the instruction required for the next action. However, different words have different functions in a sentence (e.g., modifiers convey attributes, verbs convey actions). Syntax information like dependencies and phrase structures can aid the agent to locate important parts of the instruction. Hence, in this paper, we propose a navigation agent that utilizes syntax information derived from a dependency tree to enhance alignment between the instruction and the current visual scenes. Empirically, our agent outperforms the baseline model that does not use syntax information on the Room-to-Room dataset, especially in the unseen environment. Besides, our agent achieves the new state-of-the-art on Room-Across-Room dataset, which contains instructions in 3 languages (English, Hindi, and Telugu). We also show that our agent is better at aligning instructions with the current visual information via qualitative visualizations. Code and models: https://github.com/jialuli-luka/SyntaxVLN

Motivation & Objective

• To improve cross-modal alignment between natural language instructions and visual scenes in vision-language navigation.
• To investigate whether syntactic information from dependency trees enhances grounding and navigation performance.
• To reduce word-level overfitting and improve zero-shot generalization in unseen environments.
• To develop a syntax-aware encoder that captures phrase-level structure for better instruction understanding.
• To evaluate performance across multilingual and zero-shot generalization benchmarks.

Proposed method

• The model uses a dependency parse tree to structure the natural language instruction, extracting syntactic relationships between words.
• A ChildSum Tree-LSTM is applied over the dependency tree to generate syntax-aware contextual representations of the instruction.
• The syntax-aware instruction encoding is fused with visual features through cross-attention mechanisms to improve grounding.
• The navigation policy is trained end-to-end using imitation learning, with action prediction conditioned on aligned vision-language features.
• The approach is evaluated on both the Room-to-Room (R2R) and Room-Across-Room (RxR) datasets, including multilingual settings.
• Implementation variants include mean-pooling over dependency trees and using constituency trees, to ablate the contribution of syntactic structure.

Experimental results

Research questions

• RQ1 Can dependency tree-based syntactic structure improve cross-modal alignment in vision-language navigation?
• RQ2 Does incorporating phrase-level syntactic information lead to better generalization in unseen environments?
• RQ3 How does syntax-aware representation compare to word-level attention in terms of robustness and interpretability?
• RQ4 To what extent does syntax information improve performance on multilingual instruction following tasks?
• RQ5 Is the benefit of syntax dependent on the quality of the dependency parser in low-resource languages?

Key findings

• The proposed syntax-aware model achieves a new state-of-the-art success rate on the multilingual Room-Across-Room (RxR) dataset, outperforming prior methods at the time of submission.
• On the R2R dataset, the model improves success rate by 3.5% on unseen environments, demonstrating strong zero-shot generalization.
• The model shows a 0.7% improvement on English and 0.1% on Telugu in RxR, with Hindi seeing the largest gain, suggesting parser quality impacts performance.
• Qualitative analysis confirms that the syntax-aware agent better aligns instruction phrases with visual scenes, such as identifying 'walk past the shelves' and 'out of the garage' as distinct navigation steps.
• Ablation studies show that mean-pooling over dependency trees reduces performance by ~3%, and constituency trees yield ~2% lower success rate, indicating dependency trees are more effective.
• The model reduces word-level overfitting and learns more interpretable, phrase-level alignments, as evidenced by visual attention maps.