[Paper Review] Mind the Gap: Improving Success Rate of Vision-and-Language Navigation by Revisiting Oracle Success Routes

Chongyang Zhao, Yuankai Qi+1 more|arXiv (Cornell University)|Aug 7, 2023

Multimodal Machine Learning ApplicationsComputer Science3 citations

TL;DR

This paper addresses the long-ignored gap between Success Rate (SR) and Oracle Success Rate (OSR) in Vision-and-Language Navigation by proposing a trajectory grounding approach that identifies the correct target location from pre-generated agent trajectories. Using a multi-module transformer model to learn discriminative viewpoint representations, the method improves SR by narrowing the gap from up to 9% to 4.56% on R2R, demonstrating significant gains without retraining navigation policies.

ABSTRACT

Vision-and-Language Navigation (VLN) aims to navigate to the target location by following a given instruction. Unlike existing methods focused on predicting a more accurate action at each step in navigation, in this paper, we make the first attempt to tackle a long-ignored problem in VLN: narrowing the gap between Success Rate (SR) and Oracle Success Rate (OSR). We observe a consistently large gap (up to 9%) on four state-of-the-art VLN methods across two benchmark datasets: R2R and REVERIE. The high OSR indicates the robot agent passes the target location, while the low SR suggests the agent actually fails to stop at the target location at last. Instead of predicting actions directly, we propose to mine the target location from a trajectory given by off-the-shelf VLN models. Specially, we design a multi-module transformer-based model for learning compact discriminative trajectory viewpoint representation, which is used to predict the confidence of being a target location as described in the instruction. The proposed method is evaluated on three widely-adopted datasets: R2R, REVERIE and NDH, and shows promising results, demonstrating the potential for more future research.

Motivation & Objective

• To address the persistent performance gap between Success Rate (SR) and Oracle Success Rate (OSR) in Vision-and-Language Navigation (VLN), which remains overlooked in prior work.
• To investigate why state-of-the-art VLN agents often pass the target location but fail to stop there, despite high OSR.
• To propose a method that improves SR by detecting the correct target node in a trajectory generated by off-the-shelf VLN models, rather than retraining navigation policies.
• To evaluate the effectiveness of the approach on multiple benchmarks, including R2R, REVERIE, and NDH, under both in-domain and zero-shot settings.

Proposed method

• The method formulates target location detection as a video grounding task, where the goal is to localize the target frame (viewpoint) in a trajectory based on the natural language instruction.
• A multi-module transformer architecture is proposed, consisting of an elevation transformer (E-Trans) for panoramic view processing, a spatial-temporal transformer (S-T Trans) for modeling trajectory-level dependencies, and a target selection transformer (T-Trans) for predicting target confidence scores.
• The model learns compact, discriminative representations of trajectory viewpoints by attending to relevant visual and textual cues across multiple views and time steps.
• The method is trained using a combination of focal loss and dice loss to handle class imbalance and improve localization accuracy, with ablation studies confirming the effectiveness of the loss design.
• The approach is applied as a post-processing module on trajectories from existing VLN agents, enabling correction of navigation errors without modifying the original policy.
• Additional data from 900 HM3D environments is used to pre-train the model, improving zero-shot generalization and performance on unseen splits.

Experimental results

Research questions

• RQ1 Why does a significant gap exist between SR and OSR in state-of-the-art VLN agents, despite high Oracle Success Rates?
• RQ2 Can the target location be reliably identified in a trajectory generated by a pre-trained VLN agent, even if the agent fails to stop at the correct location?
• RQ3 How effective is a multi-module transformer-based approach in detecting the correct target viewpoint from a sequence of panoramic observations and a natural language instruction?
• RQ4 What is the impact of auxiliary training data and loss functions on the performance of the target grounding model?

Key findings

• The proposed method reduces the SR-OSR gap from 7.96% to 4.56% on the R2R validation unseen split, representing a 4.3% absolute improvement.
• The elevation transformer (E-Trans) contributes the most to performance, followed by the target selection transformer (T-Trans), with the spatial-temporal transformer (S-T Trans) having the smallest but still meaningful impact.
• Using additional training data from HM3D environments increases SR by 2.17% compared to training on original data alone, and a model trained solely on HM3D data achieves 45.52% SR on R2R unseen.
• The combination of focal loss and dice loss yields the best performance, outperforming BCE, focal, and BCE+Dice, indicating that joint optimization of multiple predictions improves localization accuracy.
• Ablation studies show that hyperparameter tuning has minimal impact on performance, suggesting the method is robust to loss weighting variations.
• Qualitative results demonstrate that the method successfully corrects navigation errors in trajectories from HOP, HAMT, and DUET, enabling agents to stop at the correct target location.