[Paper Review] A Recurrent Vision-and-Language BERT for Navigation

Yicong Hong, Qi Wu+3 more|arXiv (Cornell University)|Nov 26, 2020

Multimodal Machine Learning Applications59 references40 citations

TL;DR

This paper introduces VLN ∘→BERT, a recurrent vision-and-language BERT for navigation that maintains history-aware cross-modal state to tackle partially observable VLN tasks, achieving state-of-the-art results on R2R and REVERIE by using a pre-trained V&L BERT as the navigator and enabling multi-task capabilities.

ABSTRACT

Accuracy of many visiolinguistic tasks has benefited significantly from the application of vision-and-language(V&L) BERT. However, its application for the task of vision-and-language navigation (VLN) remains limited. One reason for this is the difficulty adapting the BERT architecture to the partially observable Markov decision process present in VLN, requiring history-dependent attention and decision making. In this paper we propose a recurrent BERT model that is time-aware for use in VLN. Specifically, we equip the BERT model with a recurrent function that maintains cross-modal state information for the agent. Through extensive experiments on R2R and REVERIE we demonstrate that our model can replace more complex encoder-decoder models to achieve state-of-the-art results. Moreover, our approach can be generalised to other transformer-based architectures, supports pre-training, and is capable of solving navigation and referring expression tasks simultaneously.

Motivation & Objective

• Motivate and address VLN as a partially observable MDP requiring history-dependent decision making.
• Leverage pre-trained vision-and-language BERT models by introducing a recurrent, time-aware mechanism.
• Reduce memory consumption to enable training on standard GPUs while preserving cross-modal learning.
• Demonstrate that VLN ∘→BERT can replace complex encoder-decoder VLN architectures and support multi-task learning.

Proposed method

• Introduce a recurrent function into V&L BERT to model history-dependent state representations for VLN.
• Keep language tokens fixed after initial encoding to reduce computation while preserving instruction representation.
• Process visual observations at each step and fuse with the current state and initial language encoding through the Transformer architecture.
• Define a state refinement mechanism that matches raw textual and visual features to update the agent’s state.
• Infer action probabilities from mean visual attention weights and, for REVERIE, object grounding via attention-based scoring.
• Train with a mix of reinforcement learning (A2C) and imitation learning (cross-entropy against teacher actions).
• Initialise VLN ∘→BERT from OSCAR or PREVALENT pre-trained models to maximize transfer of VLN-relevant knowledge.

Experimental results

Research questions

• RQ1 How can a pre-trained vision-and-language BERT be adapted to handle the partial observability and temporal dependency in VLN?
• RQ2 Can a recurrent extension of V&L BERT achieve competitive navigation and referring expression tasks without external memory modules?
• RQ3 Does initializing from existing VLN-pretrained models (OSCAR, PREVALENT) improve generalization to unseen environments and instructions?
• RQ4 What are the effects of making language features static after initial encoding on training efficiency and performance?
• RQ5 Can a single VLN framework jointly handle navigation and remote referring expression grounding effectively?

Key findings

• VLN ∘→BERT achieves state-of-the-art results on R2R and REVERIE, using a recurrent V&L BERT without extra memory buffers or LSTMs.
• On R2R test unseen, the model with PREVALENT pre-training reaches 63% SR and 57% SPL, showing strong generalisation.
• On REVERIE test unseen, the model with OSCAR PRETRAINING achieves 24.62% navigation SR and 19.48% SPL, with 12.65% RGS and 10.00% RGSPL for remote grounding.
• Initialising from PREVALENT generally provides the best results across most splits and metrics, highlighting the benefit of navigation-oriented VLN pre-training for multi-task performance.
• Ablation studies show that using a single, fully fine-tuned VLN BERT with state refinement and cross-modal matching outperforms variants that re-attend language at every step, while maintaining manageable GPU memory usage.