[Paper Review] SOAT: A Scene- and Object-Aware Transformer for Vision-and-Language Navigation

Abhinav Moudgil, Arjun Majumdar+3 more|arXiv (Cornell University)|Oct 27, 2021

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes SOAT, a scene- and object-aware transformer for vision-and-language navigation that uses separate visual encoders for scene classification and object detection, integrating them via a selective attention mechanism to improve grounding of language instructions. The method achieves 1.8% absolute improvement in SPL on R2R and 3.7% in SR on RxR, with even larger gains—up to 7.9% in SR—on object-heavy instructions, demonstrating the effectiveness of vision-and-language pretraining with object-level features.

ABSTRACT

Natural language instructions for visual navigation often use scene descriptions (e.g., "bedroom") and object references (e.g., "green chairs") to provide a breadcrumb trail to a goal location. This work presents a transformer-based vision-and-language navigation (VLN) agent that uses two different visual encoders -- a scene classification network and an object detector -- which produce features that match these two distinct types of visual cues. In our method, scene features contribute high-level contextual information that supports object-level processing. With this design, our model is able to use vision-and-language pretraining (i.e., learning the alignment between images and text from large-scale web data) to substantially improve performance on the Room-to-Room (R2R) and Room-Across-Room (RxR) benchmarks. Specifically, our approach leads to improvements of 1.8% absolute in SPL on R2R and 3.7% absolute in SR on RxR. Our analysis reveals even larger gains for navigation instructions that contain six or more object references, which further suggests that our approach is better able to use object features and align them to references in the instructions.

Motivation & Objective

• To improve vision-and-language navigation (VLN) by explicitly modeling both scene-level and object-level visual cues in instructions.
• To address the domain shift between vision-and-language pretraining and VLN fine-tuning by using object detection features aligned with pretraining data.
• To design a transformer architecture that selectively uses scene features as contextual cues while focusing processing on object-level representations.
• To evaluate whether vision-and-language pretraining enables effective use of object features in VLN, especially in complex, object-rich instructions.
• To demonstrate that object-level grounding significantly improves navigation performance on challenging benchmarks like RxR, particularly for instructions with many object references.

Proposed method

• The model employs two distinct visual encoders: a Places-based CNN for scene classification and a Faster R-CNN for object detection, producing complementary visual features.
• Scene features are used as high-level contextual cues, while object features are processed through a modified transformer with a selective attention mechanism that freezes scene representations.
• The attention mechanism is restructured to allow object features to attend to scene features for context, but not vice versa, ensuring scene features remain fixed and serve as context.
• The model is fine-tuned end-to-end using vision-and-language pretraining (e.g., VLN-BERT), leveraging large-scale web data to align image and text representations.
• A view aggregation strategy selects the most relevant visual features for action prediction at each navigation step.
• The method is evaluated on the R2R and RxR benchmarks, with ablation studies isolating the impact of object features and pretraining.

Experimental results

Research questions

• RQ1 Can explicit modeling of both scene and object visual cues improve vision-and-language navigation performance?
• RQ2 Does using object detection features—aligned with vision-and-language pretraining—lead to better grounding of referring expressions in navigation instructions?
• RQ3 How does the selective attention mechanism that treats scene and object features differently affect performance compared to standard attention?
• RQ4 To what extent does vision-and-language pretraining enable effective use of object-level features in VLN?
• RQ5 Does the model show greater gains on instructions with high object reference density, indicating improved object grounding?

Key findings

• SOAT achieves a 1.8% absolute improvement in SPL on the R2R validation-unseen split compared to the VLN-BERT baseline.
• On the more challenging RxR benchmark, SOAT improves success rate (SR) by 3.7 percentage points and NDTW by 2.4 points.
• On instructions with six or more object references, SOAT achieves a 7.91% absolute improvement in SR over the VLN-BERT baseline.
• The performance gain from object features is negligible without vision-and-language pretraining, indicating that pretraining is essential for leveraging object-level representations.
• The model outperforms state-of-the-art methods on RxR for English-language instructions, particularly in object-heavy navigation scenarios.
• Ablation studies confirm that the selective attention mechanism is critical—simply adding object features without architectural changes degrades performance.