[Paper Review] A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility

Andrea Burns, Deniz Arsan+4 more|arXiv (Cornell University)|Feb 4, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

This paper introduces MoTIF, a new vision-language navigation dataset for mobile apps that includes both feasible and infeasible natural language commands, with fine-grained feasibility annotations and follow-up questions. It proposes the novel task of interactive feasibility prediction, demonstrating that visual understanding is crucial for accuracy, and benchmarks state-of-the-art methods, revealing significant generalization challenges across unseen app environments.

ABSTRACT

Vision-language navigation (VLN), in which an agent follows language instruction in a visual environment, has been studied under the premise that the input command is fully feasible in the environment. Yet in practice, a request may not be possible due to language ambiguity or environment changes. To study VLN with unknown command feasibility, we introduce a new dataset Mobile app Tasks with Iterative Feedback (MoTIF), where the goal is to complete a natural language command in a mobile app. Mobile apps provide a scalable domain to study real downstream uses of VLN methods. Moreover, mobile app commands provide instruction for interactive navigation, as they result in action sequences with state changes via clicking, typing, or swiping. MoTIF is the first to include feasibility annotations, containing both binary feasibility labels and fine-grained labels for why tasks are unsatisfiable. We further collect follow-up questions for ambiguous queries to enable research on task uncertainty resolution. Equipped with our dataset, we propose the new problem of feasibility prediction, in which a natural language instruction and multimodal app environment are used to predict command feasibility. MoTIF provides a more realistic app dataset as it contains many diverse environments, high-level goals, and longer action sequences than prior work. We evaluate interactive VLN methods using MoTIF, quantify the generalization ability of current approaches to new app environments, and measure the effect of task feasibility on navigation performance.

Motivation & Objective

• To address the gap in vision-language navigation research where commands are assumed feasible, even when real-world requests may be ambiguous or impossible.
• To create a scalable, realistic dataset for interactive vision-language navigation using mobile apps, which involve complex, state-changing actions like tapping, typing, and swiping.
• To introduce and benchmark the new problem of feasibility prediction in multimodal, interactive environments.
• To evaluate the generalization of VLN methods across unseen app environments and analyze the impact of task feasibility on navigation performance.
• To enable research on uncertainty resolution through follow-up questions for ambiguous or infeasible tasks.

Proposed method

• The MoTIF dataset collects natural language commands, app screen images, view hierarchies (backend structure), and action coordinates (clicks, taps, scrolls, text input) across multiple mobile app environments.
• Each command is labeled with a binary feasibility label and subclass annotations explaining *why* it is infeasible (e.g., missing functionality, ambiguous language, state dependency).
• Follow-up questions are collected to model human-like dialogue for resolving task uncertainty.
• A baseline model for feasibility prediction is proposed, using multimodal inputs (vision and language) to classify command feasibility.
• The paper evaluates existing VLN methods (Seq2Seq, MOCA, Seq2Act) on MoTIF for task automation, comparing performance with and without high-level goals and low-level instructions.
• Extensive ablation studies assess the role of vision, language, and app exploration in feasibility prediction and action sequence generation.

Experimental results

Research questions

• RQ1 Can vision-language models accurately predict whether a natural language command is feasible in a mobile app environment, even when the command is ambiguous or the required functionality is missing?
• RQ2 How do different input modalities—high-level goals, low-level instructions, and visual inputs—affect the performance and generalization of VLN models on unseen app environments?
• RQ3 To what extent does task feasibility impact navigation performance, and how do models fail when faced with infeasible commands?
• RQ4 Can follow-up questions effectively resolve uncertainty in ambiguous or infeasible commands, and how do they improve model robustness?
• RQ5 How well do existing VLN methods generalize to new app environments when tested on MoTIF’s diverse and challenging test split?

Key findings

• Visual inputs are essential for accurate feasibility prediction, as models relying solely on language or app icons perform worse than those using multimodal visual features.
• The best-performing model for feasibility prediction achieves 51.1% complete sequence accuracy and 21.3% grounding accuracy on unseen apps, indicating significant room for improvement.
• Models generalize significantly better to new tasks than to new app environments, highlighting a key challenge in zero-shot app generalization.
• Low-level step-by-step instructions lead to substantially higher action and grounding accuracy (e.g., 97.3% action accuracy with Seq2Act) compared to high-level goals alone.
• Even with both high-level and low-level inputs, performance degrades on unseen apps, suggesting that more in-app exploration during training is necessary for robustness.
• Prior app-specific representation learning (e.g., using app icons) does not yield the best performance, underscoring the importance of visual understanding over domain-specific features.