[Paper Review] Agent as Cerebrum, Controller as Cerebellum: Implementing an Embodied LMM-based Agent on Drones

Haoran Zhao, Fengxing Pan+2 more|arXiv (Cornell University)|Nov 25, 2023

Reinforcement Learning in Robotics6 citations

TL;DR

This paper proposes a novel 'agent as cerebrum, controller as cerebellum' architecture that leverages a Large Multimodal Model (LMM)-based agent, AeroAgent, for industrial drone applications. By decoupling high-level reasoning (agent) from real-time control (controller), and integrating it via a custom ROSchain framework, the system achieves superior performance in complex, real-world tasks like search and rescue, outperforming DRL-based agents in both simulation (Airgen) and real-world trials.

ABSTRACT

In this study, we present a novel paradigm for industrial robotic embodied agents, encapsulating an 'agent as cerebrum, controller as cerebellum' architecture. Our approach harnesses the power of Large Multimodal Models (LMMs) within an agent framework known as AeroAgent, tailored for drone technology in industrial settings. To facilitate seamless integration with robotic systems, we introduce ROSchain, a bespoke linkage framework connecting LMM-based agents to the Robot Operating System (ROS). We report findings from extensive empirical research, including simulated experiments on the Airgen and real-world case study, particularly in individual search and rescue operations. The results demonstrate AeroAgent's superior performance in comparison to existing Deep Reinforcement Learning (DRL)-based agents, highlighting the advantages of the embodied LMM in complex, real-world scenarios.

Motivation & Objective

• To address the limitations of end-to-end deep reinforcement learning (DRL) agents in industrial drones, such as poor generalization, slow inference, and sim-to-real transfer issues.
• To enable high-level reasoning and task planning in drones using Large Multimodal Models (LMMs), while offloading real-time control to established robotic controllers.
• To bridge the gap between LMM-based agents and real-world robotic systems by developing a dedicated integration framework, ROSchain, for seamless ROS communication.
• To validate the efficacy of the proposed architecture in complex, real-world industrial scenarios, particularly in dynamic and unstructured environments like search and rescue operations.

Proposed method

• Designs an agent architecture, AeroAgent, composed of a multimodal memory, an embodied plan generator, and an integrated action library to support high-level reasoning and task planning.
• Employs a decoupled control paradigm where the LMM-based agent acts as the 'cerebrum' for strategic decision-making, while the drone’s native flight controller serves as the 'cerebellum' for real-time, low-level actuation.
• Develops ROSchain, a custom middleware framework that enables secure, low-latency communication between the LMM-based agent and the Robot Operating System (ROS), ensuring compatibility with existing drone control stacks.
• Utilizes a hybrid planning mechanism: the agent generates high-level waypoints and task sequences using multimodal reasoning, which are then executed via the drone’s native control system.
• Employs few-shot learning and introspective reasoning in the agent’s memory and planning modules to adapt to novel or unseen scenarios without retraining.
• Integrates obstacle avoidance and priority management into the flight control layer, ensuring safe navigation during task execution.

Experimental results

Research questions

• RQ1 Can an LMM-based agent effectively manage complex, high-level tasks in industrial drones without direct control over low-level actuators?
• RQ2 How does the decoupled agent-controller architecture improve performance and robustness compared to end-to-end DRL-based agents in real-world drone applications?
• RQ3 To what extent does the ROSchain framework enable reliable, low-latency integration between LMM agents and ROS-based robotic systems?
• RQ4 How well does the agent generalize to new tasks and environments, particularly in unstructured, real-world settings like search and rescue?
• RQ5 What is the contribution of the agent’s multimodal memory and planning module to few-shot adaptation and task generalization?

Key findings

• AeroAgent outperformed existing DRL-based agents in both simulated (Airgen) and real-world search and rescue scenarios, demonstrating superior task completion rates and robustness.
• The ablation study confirmed that ROSchain significantly enhances system stability and communication efficiency, reducing latency and improving real-time responsiveness.
• The agent’s multimodal memory enabled effective few-shot adaptation to novel tasks, allowing successful execution without retraining or fine-tuning.
• The decoupled architecture achieved real-time control performance (sub-10ms response) suitable for dynamic flight, overcoming the latency limitations of direct LMM control.
• The system demonstrated reliable obstacle avoidance and task prioritization in real-world trials, maintaining flight stability and mission integrity.
• The framework proved effective in complex, unstructured environments, validating its potential for industrial applications such as power line inspection and autonomous drone herding.