[Paper Review] Just Ask:An Interactive Learning Framework for Vision and Language Navigation

Ta-Chung Chi, Mihail Eric+3 more|arXiv (Cornell University)|Dec 2, 2019

Multimodal Machine Learning Applications26 references6 citations

TL;DR

This paper proposes an interactive learning framework for vision-and-language navigation that enables agents to resolve ambiguities by asking humans for help during navigation. Using a reinforcement learning agent with reward shaping or a simpler confusion-based method, the agent learns when and what to ask, boosting success rates by at least 15% while asking only one question on average, and demonstrating robustness to noisy human responses and data efficiency through human-guided continual learning.

ABSTRACT

In the vision and language navigation task, the agent may encounter ambiguous situations that are hard to interpret by just relying on visual information and natural language instructions. We propose an interactive learning framework to endow the agent with the ability to ask for users' help in such situations. As part of this framework, we investigate multiple learning approaches for the agent with different levels of complexity. The simplest model-confusion-based method lets the agent ask questions based on its confusion, relying on the predefined confidence threshold of a next action prediction model. To build on this confusion-based method, the agent is expected to demonstrate more sophisticated reasoning such that it discovers the timing and locations to interact with a human. We achieve this goal using reinforcement learning (RL) with a proposed reward shaping term, which enables the agent to ask questions only when necessary. The success rate can be boosted by at least 15% with only one question asked on average during the navigation. Furthermore, we show that the RL agent is capable of adjusting dynamically to noisy human responses. Finally, we design a continual learning strategy, which can be viewed as a data augmentation method, for the agent to improve further utilizing its interaction history with a human. We demonstrate the proposed strategy is substantially more realistic and data-efficient compared to previously proposed pre-exploration techniques.

Motivation & Objective

• To address ambiguity in vision-and-language navigation where natural language instructions and visual cues are insufficient for decision-making.
• To enable agents to proactively ask humans for help in real-time, rather than relying on passive navigation or pre-collected data.
• To develop a data-efficient, realistic learning strategy that leverages human interaction history for continual improvement.
• To investigate the timing, type, and form of agent questions to optimize communication efficiency and task success.
• To improve robustness to noisy human responses and reduce reliance on pre-exploration data collection.

Proposed method

• Proposes a confusion-based agent (MC) that asks questions when its action prediction confidence falls below a threshold, mimicking human confusion.
• Introduces a reinforcement learning agent (ASA) with a reward shaping term that learns when to ask questions to maximize task success.
• Uses a human-in-the-loop interaction setup where the agent asks for the correct next action, and the human responds with the ground-truth action.
• Employs a continual learning strategy that fine-tunes the agent using interaction history collected during real user interactions.
• Applies a two-stage fine-tuning process: supervised fine-tuning on human-guided data, optionally combined with RL, to improve generalization.
• Designs a realistic evaluation protocol with disjoint and random splits of unseen environments to simulate real-world deployment.

Experimental results

Research questions

• RQ1 What is the optimal timing for an agent to ask a human for help during vision-and-language navigation?
• RQ2 How can an agent be trained to ask only necessary questions while maintaining high task success rates?
• RQ3 Can an agent adapt to noisy or inconsistent human responses during interaction?
• RQ4 How does human-guided exploration compare to pre-exploration data augmentation in terms of data efficiency and real-world applicability?
• RQ5 Can interaction history be used to improve the agent’s performance over time in new environments?

Key findings

• The reinforcement learning agent (ASA) achieves a success rate boost of at least 15% with only one question asked on average per navigation episode.
• The ASA agent dynamically adapts to noise in human responses, maintaining high success rates even when response quality degrades, unlike the confusion-based model.
• Human-guided exploration outperforms pre-exploration data augmentation by 5% (0.554 vs. 0.504) on a disjoint test split, demonstrating superior data efficiency.
• Even when fine-tuning is restricted to supervised learning only (without RL), the human-guided method still outperforms the baseline by 3% (0.514 vs. 0.483).
• The method achieves better performance with significantly less data than pre-exploration techniques, as shown in data efficiency curves where human-guided exploration reaches peak performance with far fewer trajectories.
• The MC agent maintains consistent performance across settings, but the ASA agent shows greater adaptability and exploration behavior due to end-to-end training.