[Paper Review] Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds

Sipeng Zheng, Jiazheng Liu+2 more|arXiv (Cornell University)|Oct 20, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

Steve-Eye is a large multimodal model that equips LLM-based embodied agents with visual perception in open-world environments like Minecraft by integrating a visual encoder with a pre-trained language model. Trained on 850K instruction-following pairs curated via ChatGPT, it achieves state-of-the-art performance in multimodal perception, foundational knowledge grounding, and skill planning, outperforming baselines on three open-world benchmarks with success rates up to 73% on complex tasks.

ABSTRACT

Recent studies have presented compelling evidence that large language models (LLMs) can equip embodied agents with the self-driven capability to interact with the world, which marks an initial step toward versatile robotics. However, these efforts tend to overlook the visual richness of open worlds, rendering the entire interactive process akin to "a blindfolded text-based game." Consequently, LLM-based agents frequently encounter challenges in intuitively comprehending their surroundings and producing responses that are easy to understand. In this paper, we propose Steve-Eye, an end-to-end trained large multimodal model designed to address this limitation. Steve-Eye integrates the LLM with a visual encoder which enables it to process visual-text inputs and generate multimodal feedback. In addition, we use a semi-automatic strategy to collect an extensive dataset comprising 850K open-world instruction pairs, empowering our model to encompass three essential functions for an agent: multimodal perception, foundational knowledge base, and skill prediction and planning. Lastly, we develop three open-world evaluation benchmarks, then carry out extensive experiments from a wide range of perspectives to validate our model's capability to strategically act and plan. Codes and datasets will be released.

Motivation & Objective

• To address the limitation of text-only LLM-based agents in open-world environments, which lack visual understanding and produce unintuitive, context-ambiguous responses.
• To develop a multimodal agent capable of perceiving visual and textual inputs, grounding its reasoning in real-world context, and generating actionable plans.
• To create a large-scale, high-quality instruction dataset that enables end-to-end training for multimodal perception, knowledge base integration, and skill planning.
• To validate the model’s capabilities through comprehensive benchmarks in open-ended environments like Minecraft, demonstrating strategic planning and execution.

Proposed method

• The Steve-Eye model combines a vision encoder (e.g., CLIP-based) with a pre-trained LLM to process multimodal inputs (image + text) and generate multimodal outputs.
• A semi-automatic data collection pipeline uses ChatGPT to generate 850K instruction-following data pairs, including visual observations, textual context, and expert-reasoned responses.
• The model is fine-tuned end-to-end on this dataset to learn three core functions: multimodal perception, foundational knowledge grounding, and skill prediction and planning.
• Evaluation is conducted on three open-world benchmarks: skill prediction, skill planning, and multimodal perception, using Minecraft as the primary testbed.
• The model supports self-driven operation by replacing rule-based game state judgments with learned skill prediction, enabling autonomous execution.
• A variant, Steve-Eye-auto, replaces external game judgment with the model’s own skill prediction, enabling fully autonomous interaction.

Experimental results

Research questions

• RQ1 Can a large multimodal model effectively integrate visual and textual inputs to improve an agent’s environmental understanding in open-world environments?
• RQ2 To what extent can a fine-tuned multimodal model outperform text-only LLMs in skill planning and execution within complex, open-ended environments?
• RQ3 How effective is a semi-automatically curated instruction dataset in enabling multimodal perception, knowledge grounding, and planning in embodied agents?
• RQ4 Can a multimodal agent replace rule-based game state evaluation with learned, self-predicted skill execution, and how does performance degrade in complex tasks?

Key findings

• Steve-Eye achieved a 92.1% recall and 84.2% accuracy on the skill prediction benchmark using the 13b parameter version, significantly outperforming baseline models.
• On the skill planning benchmark, Steve-Eye achieved a mean success rate of 73% across 24 complex tasks, surpassing MineAgent (0.46) and GPT Assistant (0.57) in key metrics.
• Steve-Eye-auto, which replaces rule-based game judgment with model-predicted skills, achieved a 70% success rate on the most challenging task, demonstrating strong self-driven capability.
• The model’s performance degradation in long, complex tasks was observed but remained significantly better than baselines, indicating robustness in extended reasoning.
• Qualitative analysis in Figure 5 shows that Steve-Eye generates coherent, step-by-step skill plans that align with the task goal, such as crafting a stone axe from a wooden pickaxe.
• The 850K instruction dataset enabled the model to learn nuanced visual-textual reasoning, common-sense knowledge, and sequential planning, essential for open-world autonomy.