[Paper Review] MLANet: Multi-Level Attention Network with Sub-instruction for Continuous Vision-and-Language Navigation

Zongtao He, Liuyi Wang+4 more|arXiv (Cornell University)|Mar 2, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes MLANet, a Multi-Level Attention Network with sub-instruction supervision for continuous vision-and-language navigation. It introduces an annotation-free Fast Sub-instruction Algorithm (FSA) to generate sub-instructions efficiently, a Multi-Level Attention (MLA) module to fuse low-level and high-level semantics, and a Peak Attention Loss (PAL) to dynamically focus on the current sub-instruction, achieving state-of-the-art performance with 31% SPL improvement on val-unseen and 28% on test splits.

ABSTRACT

Vision-and-Language Navigation (VLN) aims to develop intelligent agents to navigate in unseen environments only through language and vision supervision. In the recently proposed continuous settings (continuous VLN), the agent must act in a free 3D space and faces tougher challenges like real-time execution, complex instruction understanding, and long action sequence prediction. For a better performance in continuous VLN, we design a multi-level instruction understanding procedure and propose a novel model, Multi-Level Attention Network (MLANet). The first step of MLANet is to generate sub-instructions efficiently. We design a Fast Sub-instruction Algorithm (FSA) to segment the raw instruction into sub-instructions and generate a new sub-instruction dataset named ``FSASub". FSA is annotation-free and faster than the current method by 70 times, thus fitting the real-time requirement in continuous VLN. To solve the complex instruction understanding problem, MLANet needs a global perception of the instruction and observations. We propose a Multi-Level Attention (MLA) module to fuse vision, low-level semantics, and high-level semantics, which produce features containing a dynamic and global comprehension of the task. MLA also mitigates the adverse effects of noise words, thus ensuring a robust understanding of the instruction. To correctly predict actions in long trajectories, MLANet needs to focus on what sub-instruction is being executed every step. We propose a Peak Attention Loss (PAL) to improve the flexible and adaptive selection of the current sub-instruction. PAL benefits the navigation agent by concentrating its attention on the local information, thus helping the agent predict the most appropriate actions. We train and test MLANet in the standard benchmark. Experiment results show MLANet outperforms baselines by a significant margin.

Motivation & Objective

• To address the challenges of real-time execution, complex instruction understanding, and long-horizon action prediction in continuous vision-and-language navigation (VLN).
• To improve instruction comprehension by introducing sub-instructions as intermediate semantic units with higher granularity than individual words.
• To enable dynamic, adaptive attention over sub-instructions during navigation via a novel loss function.
• To develop an efficient, annotation-free sub-instruction generation method suitable for real-time deployment.

Proposed method

• Proposes a Fast Sub-instruction Algorithm (FSA), an efficient, annotation-free method that segments raw instructions into sub-instructions 70× faster than prior deep learning-based approaches.
• Introduces a Multi-Level Attention (MLA) module that fuses visual features, low-level word-level semantics, and high-level sub-instruction semantics into a dynamic, globally coherent representation.
• Employs a Peak Attention Loss (PAL) to shape attention scores such that only one sub-instruction is actively attended at each step, enhancing local goal awareness.
• Uses a Gaussian, constant, linear, quadratic, or cubic attention focusing ratio (σ) to control the temporal attention peak, with Gaussian yielding optimal performance.
• Trains and evaluates MLANet on standard continuous VLN benchmarks using the R2R and RxR datasets with full test splits.
• Releases a new sub-instruction dataset, FSASub, generated via FSA, for public use in future research.

Experimental results

Research questions

• RQ1 Can sub-instruction segmentation improve instruction understanding and navigation performance in continuous VLN?
• RQ2 How does multi-level attention—fusing word-level and sub-instruction-level semantics—enhance agent perception and robustness?
• RQ3 Can an unsupervised attention loss like PAL effectively guide the agent to focus on the correct sub-instruction at each step?
• RQ4 How efficient is the proposed FSA algorithm compared to existing sub-instruction generation methods, and can it support real-time navigation?
• RQ5 Does the model generalize well to unseen environments when trained on seen environments?

Key findings

• MLANet achieves a 31% improvement in SPL on the val-unseen split compared to baseline models, demonstrating strong generalization to unseen environments.
• On the test split, MLANet achieves a 28% SPL improvement, confirming its effectiveness in real-world-like navigation scenarios.
• The Peak Attention Loss (PAL) with a Gaussian focusing ratio (σ = 0.6) yields the best performance, with 30.1% success rate and 28.4% SPL on val-unseen.
• The FSA algorithm generates sub-instructions 70× faster than prior methods, making it suitable for real-time continuous VLN applications.
• Qualitative analysis shows that the model’s attention shifts appropriately across sub-instructions over time, with a clear diagonal pattern in attention heatmaps, indicating correct temporal alignment.
• Failure cases reveal sensitivity to rare visual or linguistic cues (e.g., 'lit chandelier'), suggesting limitations in robustness under uncommon observations.