[Paper Review] A Survey on Robotics with Foundation Models: toward Embodied AI

Zhiyuan Xu, Kun Wu+5 more|arXiv (Cornell University)|Feb 4, 2024

Modular Robots and Swarm Intelligence4 citations

TL;DR

This survey explores the integration of foundation models—such as large language, vision, and vision-language models—into robotics for embodied AI, focusing on high-level planning and low-level control in autonomous manipulation. It demonstrates how these models enable few-shot or zero-shot generalization across complex, long-horizon tasks through unified planning-control frameworks, while identifying key challenges in efficiency, safety, and data augmentation.

ABSTRACT

While the exploration for embodied AI has spanned multiple decades, it remains a persistent challenge to endow agents with human-level intelligence, including perception, learning, reasoning, decision-making, control, and generalization capabilities, so that they can perform general-purpose tasks in open, unstructured, and dynamic environments. Recent advances in computer vision, natural language processing, and multi-modality learning have shown that the foundation models have superhuman capabilities for specific tasks. They not only provide a solid cornerstone for integrating basic modules into embodied AI systems but also shed light on how to scale up robot learning from a methodological perspective. This survey aims to provide a comprehensive and up-to-date overview of foundation models in robotics, focusing on autonomous manipulation and encompassing high-level planning and low-level control. Moreover, we showcase their commonly used datasets, simulators, and benchmarks. Importantly, we emphasize the critical challenges intrinsic to this field and delineate potential avenues for future research, contributing to advancing the frontier of academic and industrial discourse.

Motivation & Objective

• To provide a comprehensive, up-to-date overview of foundation models in robotics, particularly for embodied AI in autonomous manipulation.
• To categorize and analyze recent approaches that apply foundation models to high-level planning and low-level control in robotic systems.
• To survey commonly used datasets, simulators, and benchmarks in the field to support reproducible research and practical application.
• To identify critical challenges such as computational efficiency, safety, and data scarcity in real-world deployment.
• To outline future research directions for advancing general-purpose, generalizable, and robust embodied AI systems.

Proposed method

• Categorizes foundation model applications in robotics into two main streams: high-level planning (e.g., interpreting natural language commands and generating task plans) and low-level control (e.g., generating precise motor commands).
• Reviews the use of vision-language models (VLMs) and large language models (LLMs) to interpret user instructions and perceive environments, enabling reasoning over world knowledge.
• Analyzes structured planning formats (e.g., PDDL) versus unstructured formats (e.g., natural language or code), evaluating trade-offs in expressiveness and executability.
• Examines end-to-end training of large robotic foundation models that unify planning and control, mimicking the cerebrum-cerebellum collaboration for improved task execution.
• Investigates data augmentation techniques using VLMs and diffusion models (e.g., Imagen Editor) to generate semantically meaningful image edits without destroying task-relevant content.
• Evaluates efficiency techniques such as LoRA fine-tuning and model compression for deploying large foundation models on edge devices, addressing computational and privacy constraints.

Figure 1 : Taxonomy of foundation models for high-level planning and their properties.

Experimental results

Research questions

• RQ1 How can foundation models be effectively leveraged to enable zero-shot or few-shot generalization in long-horizon robotic manipulation tasks?
• RQ2 What are the trade-offs between structured (e.g., PDDL) and unstructured (e.g., natural language) planning formats in terms of expressiveness, execution feasibility, and model performance?
• RQ3 How can foundation models be integrated into both high-level planning and low-level control to create unified, scalable robotic agents?
• RQ4 What are the most effective data augmentation strategies that preserve semantic meaning while improving policy generalization in robotic learning?
• RQ5 What are the key challenges in deploying foundation models on resource-constrained robotic systems, and how can they be mitigated through efficiency and safety mechanisms?

Key findings

• Foundation models such as PaLM-E and RT-1 demonstrate strong few-shot and zero-shot capabilities in robotic manipulation, enabling generalization across diverse tasks without task-specific fine-tuning.
• VLM-based data augmentation methods like ROSIE improve policy generalization on hundreds of tasks by generating semantically consistent image edits using textual instructions and diffusion models.
• LoRA and other parameter-efficient fine-tuning techniques significantly reduce the computational cost of adapting large foundation models to robotic tasks.
• Despite progress, current foundation models still struggle with out-of-distribution (OOD) states and lack robust error recovery strategies, highlighting a key gap in real-world safety.
• Deploying large foundation models on edge devices remains challenging due to high inference costs and privacy concerns, necessitating model compression and lightweight architectures.
• Safety remains a major hurdle: existing theoretical guarantees (e.g., Lyapunov-based methods) are limited to low-dimensional, idealized environments and do not generalize to high-dimensional inputs like images.