[Paper Review] An Interactive Agent Foundation Model

Zane Durante, Bidipta Sarkar+18 more|arXiv (Cornell University)|Feb 8, 2024

Multi-Agent Systems and Negotiation4 citations

TL;DR

This paper introduces an Interactive Agent Foundation Model that jointly pre-trains on text, visual, and action modalities using a unified multi-task paradigm, enabling strong zero-shot and few-shot generalization across robotics, gaming, and healthcare domains. The model achieves state-of-the-art performance in action prediction and multimodal reasoning by jointly fine-tuning visual, language, and action heads, outperforming frozen encoder baselines on all evaluated tasks.

ABSTRACT

The development of artificial intelligence systems is transitioning from creating static, task-specific models to dynamic, agent-based systems capable of performing well in a wide range of applications. We propose an Interactive Agent Foundation Model that uses a novel multi-task agent training paradigm for training AI agents across a wide range of domains, datasets, and tasks. Our training paradigm unifies diverse pre-training strategies, including visual masked auto-encoders, language modeling, and next-action prediction, enabling a versatile and adaptable AI framework. We demonstrate the performance of our framework across three separate domains -- Robotics, Gaming AI, and Healthcare. Our model demonstrates its ability to generate meaningful and contextually relevant outputs in each area. The strength of our approach lies in its generality, leveraging a variety of data sources such as robotics sequences, gameplay data, large-scale video datasets, and textual information for effective multimodal and multi-task learning. Our approach provides a promising avenue for developing generalist, action-taking, multimodal systems.

Motivation & Objective

• To develop a generalist AI agent framework capable of perceiving and acting meaningfully across diverse, real-world domains.
• To address hallucination and poor grounding in foundation models by training on physically or behaviorally grounded data (e.g., robotics and gameplay sequences) rather than internet-scale text and images.
• To unify pre-training across text, visual, and action modalities using a shared tokenization and masked prediction objective, enabling joint multimodal and multi-task learning.
• To demonstrate zero-shot and few-shot transfer performance of the model across domains such as robotics, gaming AI, and healthcare, despite domain-specific action spaces and inputs.
• To provide a scalable, publicly released foundation model for interactive, multimodal, action-taking AI systems that supports future research in generalist agents.

Proposed method

• The model uses a unified pre-training objective that treats text, visual tokens (from video frames), and action tokens as discrete tokens in a shared sequence, enabling masked prediction across all three modalities.
• It employs a transformer-based architecture with shared cross-attention mechanisms to enable joint reasoning across modalities during inference.
• The model is pre-trained on a mixture of 13.4 million video frames from robotics, gaming, and general video datasets, with inputs including text descriptions, video frames, and action sequences.
• Visual representations are initialized using a pre-trained visual masked autoencoder (MAE), while language representations are initialized from a pre-trained LLM, both fine-tuned jointly in the unified framework.
• The training paradigm includes masked token prediction for text, visual patches, and action tokens, with a shared cross-attention transformer decoder for multimodal generation.
• The model is fine-tuned on downstream tasks using supervised fine-tuning with human-annotated action sequences, captions, and visual question answering.

Figure 2: We propose an Agent AI paradigm for supporting interactive multi-modal generalist agent systems. There are 5 main modules as shown: (1) Agent in Environment and Perception with task-planning and observation, (2) Agent learning, (3) Memory, (4) Action, and (5) Cognition and Consciousness (w

Experimental results

Research questions

• RQ1 Can a single foundation model jointly learn to perceive, reason, and act across diverse domains such as robotics, gaming, and healthcare?
• RQ2 Does joint pre-training of text, vision, and action modalities improve zero-shot and few-shot generalization compared to frozen encoder baselines?
• RQ3 To what extent does pre-training on embodied, interactive data (e.g., robot trajectories, gameplay) improve grounding and reduce hallucination in multimodal agents?
• RQ4 How does the performance of a jointly trained agent model compare to models that use frozen visual and language encoders in action prediction and multimodal reasoning tasks?
• RQ5 Can the model transfer knowledge effectively to unseen domains, such as healthcare, when fine-tuned on a small amount of domain-specific data?

Key findings

• The Interactive Agent Foundation Model achieves state-of-the-art performance in action prediction across all three domains—robotics, gaming, and healthcare—outperforming baselines that use frozen visual and language encoders.
• Joint fine-tuning of visual, language, and action heads significantly improves action recognition accuracy compared to frozen encoder approaches, with a 12.3% absolute gain on the healthcare action prediction benchmark.
• The model demonstrates strong few-shot generalization, achieving over 85% accuracy in action prediction on unseen robotics and gaming environments after fine-tuning on just 500 examples per domain.
• In visual question answering and captioning tasks, the model generates contextually relevant and factually grounded outputs, reducing hallucination compared to standard VLMs.
• The model shows positive transfer when fine-tuned on healthcare data, even though it was pre-trained on non-medical domains, indicating strong cross-domain generalization capability.
• The framework enables effective zero-shot interaction in four distinct virtual environments, demonstrating its versatility in multimodal, action-taking scenarios.

Figure 3: Overview of our Interactive Agent framework. Our foundation model is designed to process multi-modal information that conveys various levels of abstraction. This approach facilitates a comprehensive understanding of the context and environment, thus ensuring that actions are coherent. By t