[Paper Review] Transferable Representation Learning in Vision-and-Language Navigation

Haoshuo Huang, Vihan Jain+5 more|arXiv (Cornell University)|Aug 9, 2019

Multimodal Machine Learning Applications46 references5 citations

TL;DR

This paper proposes a transferable representation learning framework for vision-and-language navigation (VLN) by pre-training vision and language encoders on two auxiliary tasks—Cross-Modal Alignment (CMA) and Next Visual Scene (NVS)—using self-supervised negative mining. The domain-adapted representations significantly improve navigation performance, with the RCM agent achieving a 5% absolute improvement in SPL over prior state-of-the-art models.

ABSTRACT

Vision-and-Language Navigation (VLN) tasks such as Room-to-Room (R2R) require machine agents to interpret natural language instructions and learn to act in visually realistic environments to achieve navigation goals. The overall task requires competence in several perception problems: successful agents combine spatio-temporal, vision and language understanding to produce appropriate action sequences. Our approach adapts pre-trained vision and language representations to relevant in-domain tasks making them more effective for VLN. Specifically, the representations are adapted to solve both a cross-modal sequence alignment and sequence coherence task. In the sequence alignment task, the model determines whether an instruction corresponds to a sequence of visual frames. In the sequence coherence task, the model determines whether the perceptual sequences are predictive sequentially in the instruction-conditioned latent space. By transferring the domain-adapted representations, we improve competitive agents in R2R as measured by the success rate weighted by path length (SPL) metric.

Motivation & Objective

• To address the domain shift between pre-trained vision and language models and the downstream VLN task in realistic 3D environments.
• To reduce reliance on expensive human-annotated instruction-path pairs by creating high-quality, self-supervised auxiliary tasks.
• To improve the generalization and success of navigation agents by adapting pre-trained representations to in-domain VLN distributions.
• To enable better initialization for agents like Speaker-Follower and Reinforced Cross-Modal (RCM), enhancing their performance on both seen and unseen environments.

Proposed method

• Pre-train vision and language encoders jointly on two auxiliary tasks: Cross-Modal Alignment (CMA) and Next Visual Scene (NVS) prediction.
• Use contrastive learning with negative mining to train the NVS task, predicting future visual features in the instruction-conditioned latent space.
• Train the CMA task to determine if a given instruction matches a sequence of visual frames, using similarity scores based on cross-attention.
• Combine losses from both tasks with a weighted sum: αℒ_alignment + (1−α)ℒ_coherence, optimizing for joint representation learning.
• Transfer the fine-tuned visual and language encoders as feature extractors to downstream navigation agents, such as RCM and Speaker-Follower.
• Apply the adapted representations to improve agent policy learning via imitation learning and multi-reward reinforcement learning.

Experimental results

Research questions

• RQ1 Can self-supervised auxiliary tasks improve the transferability of pre-trained vision and language representations to vision-and-language navigation?
• RQ2 How does joint pre-training on CMA and NVS compare to pre-training on either task alone in terms of downstream navigation performance?
• RQ3 To what extent do domain-adapted representations improve generalization on unseen environments in VLN?
• RQ4 Can the learned representations enhance the performance of existing state-of-the-art agents like RCM and Speaker-Follower?
• RQ5 Does the use of negative mining for auxiliary tasks enable effective data augmentation without requiring human annotations?

Key findings

• The RCM agent initialized with representations pre-trained on both CMA and NVS achieved a 5% absolute improvement in SPL over the previous state-of-the-art.
• Joint pre-training on CMA and NVS led to a 11–12% improvement in success rate (SR) on both seen and unseen R2R validation splits compared to random or single-task initialization.
• When only NVS was pre-trained, performance dropped, indicating that cross-modal alignment is essential for effective navigation representation learning.
• Warm-starting both the visual and language encoders with domain-adapted representations yielded the highest SPL (7%+ improvement) on the unseen set, demonstrating complementary benefits.
• The ablation study confirmed that CMA and NVS learn complementary representations: CMA enhances cross-modal alignment, while NVS improves visual sequence modeling.
• The scoring model trained on CMA and NVS also enabled automatic segmentation of long instruction-path sequences, suggesting potential for curriculum learning.