[Paper Review] Scene-Intuitive Agent for Remote Embodied Visual Grounding

Xiangru Lin, Guanbin Li+1 more|arXiv (Cornell University)|Mar 24, 2021

Multimodal Machine Learning Applications46 references4 citations

TL;DR

This paper proposes a scene-intuitive agent for remote embodied visual grounding in the REVERIE task, using a two-stage training pipeline: first pre-training on scene and object grounding tasks to learn high-level instruction understanding, then employing a memory-augmented attentive action decoder to generate navigation policies. The method achieves state-of-the-art performance, improving SPL by 11.58% on Val Seen and 9.09% on Val Unseen compared to prior work.

ABSTRACT

Humans learn from life events to form intuitions towards the understanding of visual environments and languages. Envision that you are instructed by a high-level instruction, "Go to the bathroom in the master bedroom and replace the blue towel on the left wall", what would you possibly do to carry out the task? Intuitively, we comprehend the semantics of the instruction to form an overview of where a bathroom is and what a blue towel is in mind; then, we navigate to the target location by consistently matching the bathroom appearance in mind with the current scene. In this paper, we present an agent that mimics such human behaviors. Specifically, we focus on the Remote Embodied Visual Referring Expression in Real Indoor Environments task, called REVERIE, where an agent is asked to correctly localize a remote target object specified by a concise high-level natural language instruction, and propose a two-stage training pipeline. In the first stage, we pretrain the agent with two cross-modal alignment sub-tasks, namely the Scene Grounding task and the Object Grounding task. The agent learns where to stop in the Scene Grounding task and what to attend to in the Object Grounding task respectively. Then, to generate action sequences, we propose a memory-augmented attentive action decoder to smoothly fuse the pre-trained vision and language representations with the agent's past memory experiences. Without bells and whistles, experimental results show that our method outperforms previous state-of-the-art(SOTA) significantly, demonstrating the effectiveness of our method.

Motivation & Objective

• To address the challenge of navigating and localizing remote objects using high-level natural language instructions in real indoor environments.
• To design an agent that mimics human intuition by reasoning about 'where' to go and 'what' to attend to, based on commonsense knowledge and visual context.
• To improve generalization and navigation efficiency in unseen environments, overcoming limitations of existing models trained on fine-grained instructions.
• To develop a two-stage training framework that pre-trains vision-language alignment before end-to-end policy learning.

Proposed method

• Proposes a two-stage training pipeline: pre-training on Scene Grounding and Object Grounding tasks, followed by action decoding with a memory-augmented decoder.
• Introduces the Scene Grounding task to identify the correct viewpoint aligned with a high-level instruction, using cross-modal contrastive learning.
• Introduces the Object Grounding task to localize the target object among candidates at a given viewpoint, enhancing attention to relevant visual features.
• Employs a memory-augmented attentive action decoder that fuses visual features, language representations, and recurrent memory states to generate action sequences.
• Uses a multi-task loss combining cross-entropy, policy gradient, and reinforcement learning objectives to optimize navigation and grounding jointly.
• Leverages ViLBERT-based encoders for joint vision-language representation learning, with logit fusion to improve grounding accuracy.

Experimental results

Research questions

• RQ1 Can pre-training on scene and object grounding tasks improve zero-shot generalization in high-level instruction-based navigation?
• RQ2 How does integrating memory-augmented attention improve long-horizon navigation under ambiguous, high-level instructions?
• RQ3 To what extent does the proposed method outperform existing SOTA models in terms of success rate and path efficiency on the REVERIE benchmark?
• RQ4 Does the fusion of grounded visual and linguistic representations with memory states lead to more robust and accurate target localization?

Key findings

• The proposed method achieves a 11.58% improvement in SPL on the Val Seen split and a 9.09% improvement on Val Unseen compared to previous SOTA.
• For RG SPL, the method improves by 12.99% on Val Seen and 6.89% on Val Unseen, indicating better localization accuracy and navigation success.
• The Object Grounding model outperforms prior SOTA by over 10% in accuracy, demonstrating strong cross-modal alignment.
• The Scene Grounding model achieves high accuracy in identifying the correct target viewpoint, validating its ability to reason about high-level spatial instructions.
• The memory-augmented action decoder reduces average navigation trajectory length while maintaining high success rates, especially on long paths.
• Experiments show that summing the guidance from multiple heads (g_{sg}, g_{og}, and l_{τ}) improves robustness and reduces bias in target search.