[论文解读] A Recurrent Vision-and-Language BERT for Navigation

Yicong Hong, Qi Wu+3 more|arXiv (Cornell University)|Nov 26, 2020

Multimodal Machine Learning Applications参考文献 59被引用 40

一句话总结

这篇论文引入 VLN ∘→BERT，一种用于导航的循环视觉-语言 BERT，能够维持历史感知的跨模态状态以应对部分可观测的 VLN 任务，在使用一个预训练的 V&L BERT 作为导航器并实现多任务能力方面，在 R2R 和 REVERIE 上取得最先进的结果。

ABSTRACT

Accuracy of many visiolinguistic tasks has benefited significantly from the application of vision-and-language(V&L) BERT. However, its application for the task of vision-and-language navigation (VLN) remains limited. One reason for this is the difficulty adapting the BERT architecture to the partially observable Markov decision process present in VLN, requiring history-dependent attention and decision making. In this paper we propose a recurrent BERT model that is time-aware for use in VLN. Specifically, we equip the BERT model with a recurrent function that maintains cross-modal state information for the agent. Through extensive experiments on R2R and REVERIE we demonstrate that our model can replace more complex encoder-decoder models to achieve state-of-the-art results. Moreover, our approach can be generalised to other transformer-based architectures, supports pre-training, and is capable of solving navigation and referring expression tasks simultaneously.

研究动机与目标

• Motivate and address VLN as a partially observable MDP requiring history-dependent decision making.
• Leverage pre-trained vision-and-language BERT models by introducing a recurrent, time-aware mechanism.
• Reduce memory consumption to enable training on standard GPUs while preserving cross-modal learning.
• Demonstrate that VLN ∘→BERT can replace complex encoder-decoder VLN architectures and support multi-task learning.

提出的方法

• Introduce a recurrent function into V&L BERT to model history-dependent state representations for VLN.
• Keep language tokens fixed after initial encoding to reduce computation while preserving instruction representation.
• Process visual observations at each step and fuse with the current state and initial language encoding through the Transformer architecture.
• Define a state refinement mechanism that matches raw textual and visual features to update the agent’s state.
• Infer action probabilities from mean visual attention weights and, for REVERIE, object grounding via attention-based scoring.
• Train with a mix of reinforcement learning (A2C) and imitation learning (cross-entropy against teacher actions).
• Initialise VLN ∘→BERT from OSCAR or PREVALENT pre-trained models to maximize transfer of VLN-relevant knowledge.

实验结果

研究问题

• RQ1 How can a pre-trained vision-and-language BERT be adapted to handle the partial observability and temporal dependency in VLN?
• RQ2 Can a recurrent extension of V&L BERT achieve competitive navigation and referring expression tasks without external memory modules?
• RQ3 Does initializing from existing VLN-pretrained models (OSCAR, PREVALENT) improve generalization to unseen environments and instructions?
• RQ4 What are the effects of making language features static after initial encoding on training efficiency and performance?
• RQ5 Can a single VLN framework jointly handle navigation and remote referring expression grounding effectively?

主要发现

• VLN ∘→BERT achieves state-of-the-art results on R2R and REVERIE, using a recurrent V&L BERT without extra memory buffers or LSTMs.
• On R2R test unseen, the model with PREVALENT pre-training reaches 63% SR and 57% SPL, showing strong generalisation.
• On REVERIE test unseen, the model with OSCAR PRETRAINING achieves 24.62% navigation SR and 19.48% SPL, with 12.65% RGS and 10.00% RGSPL for remote grounding.
• Initialising from PREVALENT generally provides the best results across most splits and metrics, highlighting the benefit of navigation-oriented VLN pre-training for multi-task performance.
• Ablation studies show that using a single, fully fine-tuned VLN BERT with state refinement and cross-modal matching outperforms variants that re-attend language at every step, while maintaining manageable GPU memory usage.