[Paper Review] VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View

Raphael Schumann, Wanrong Zhu+4 more|arXiv (Cornell University)|Jul 12, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

VELMA is a novel LLM-based agent for vision-and-language navigation in Street View that uses verbalized visual observations and trajectory history as context for action prediction. By extracting landmarks from instructions and using CLIP to verify their visibility, VELMA achieves state-of-the-art performance with a 25% relative improvement in task completion after fine-tuning on two datasets.

ABSTRACT

Incremental decision making in real-world environments is one of the most challenging tasks in embodied artificial intelligence. One particularly demanding scenario is Vision and Language Navigation~(VLN) which requires visual and natural language understanding as well as spatial and temporal reasoning capabilities. The embodied agent needs to ground its understanding of navigation instructions in observations of a real-world environment like Street View. Despite the impressive results of LLMs in other research areas, it is an ongoing problem of how to best connect them with an interactive visual environment. In this work, we propose VELMA, an embodied LLM agent that uses a verbalization of the trajectory and of visual environment observations as contextual prompt for the next action. Visual information is verbalized by a pipeline that extracts landmarks from the human written navigation instructions and uses CLIP to determine their visibility in the current panorama view. We show that VELMA is able to successfully follow navigation instructions in Street View with only two in-context examples. We further finetune the LLM agent on a few thousand examples and achieve 25%-30% relative improvement in task completion over the previous state-of-the-art for two datasets.

Motivation & Objective

• To enable large language models (LLMs) to perform incremental decision-making in real-world, large-scale urban environments like Street View.
• To address the challenge of grounding natural language navigation instructions in dynamic, real-world visual observations.
• To improve few-shot and fine-tuned performance in vision-and-language navigation (VLN) by leveraging verbalized environment states and trajectory context.
• To resolve limitations in the Touchdown environment, such as incorrect action alignments at intersections and lack of intuitive actions like TURN_AROUND.
• To demonstrate that verbalization-based embodiment of LLMs can achieve strong performance in complex, real-world VLN tasks without end-to-end training.

Proposed method

• VELMA uses a verbalization-based prompting strategy where the LLM is prompted with a natural language sequence containing the task description, navigation instructions, past actions, and verbalized visual observations.
• Visual observations are verbalized via a pipeline that extracts landmarks from human-written instructions and uses CLIP to compute similarity scores between landmarks and current panorama views to determine visibility.
• The agent performs incremental decision-making by predicting the next action (e.g., FORWARD, TURN_LEFT, TURN_AROUND) at each step based on the full context prompt.
• A novel training procedure called Response-Based Learning (RBL) combines teacher forcing and student forcing with a mixing ratio λ=0.5 to improve policy learning and generalization.
• When the agent fails to reach the goal, an oracle computes the optimal counterfactual action sequence to provide corrective supervision.
• The method includes environment modifications: fixing intersection misalignments and adding the TURN_AROUND action to improve navigation realism and performance.

Experimental results

Research questions

• RQ1 Can LLMs be effectively embodied in real-world, large-scale urban environments like Street View using verbalized context instead of end-to-end training?
• RQ2 How effective is verbalization of visual observations (via landmark detection and CLIP-based visibility scoring) in improving LLM-based navigation performance?
• RQ3 Can few-shot in-context learning with only two examples achieve competitive performance in urban VLN, and how does fine-tuning improve it?
• RQ4 What impact do environment-level modifications—such as adding TURN_AROUND and fixing intersection issues—have on agent performance and training stability?
• RQ5 How does the choice of vision encoder (CLIP vs. OpenCLIP) affect landmark detection and overall task success?

Key findings

• VELMA achieves a 23% task completion rate in few-shot in-context learning on the Map2seq dataset and 10% on Touchdown, demonstrating strong few-shot generalization with only two few-shot examples.
• After fine-tuning on the full training set, VELMA achieves 26% task completion on Touchdown and 47% on Map2seq, representing a 25% relative improvement over the previous state-of-the-art.
• The use of OpenCLIP for landmark scoring outperforms the original CLIP model, indicating that improved vision encoders directly benefit the agent’s performance.
• Removing landmark information from the prompt significantly degrades performance, especially on Touchdown, confirming the importance of visual grounding in the context.
• The addition of the TURN_AROUND action and correction of intersection misalignments in the environment significantly improved the agent’s ability to handle complex stopping conditions.
• The agent learned to handle ambiguous stopping instructions—such as "stop before Y" or "stop at X, but not past Y"—by walking past uncertain landmarks and turning back when visible, demonstrating effective spatial reasoning.