[Paper Review] RobotGPT: Robot Manipulation Learning from ChatGPT

Yixiang Jin, Dingzhe Li+6 more|arXiv (Cornell University)|Dec 3, 2023

Artificial Intelligence in Healthcare and Education4 citations

TL;DR

RobotGPT proposes a stable, reliable framework that leverages ChatGPT's problem-solving capabilities to train a dedicated robot agent for manipulation tasks, avoiding the instability of direct code execution. By using a multi-agent prompting system with self-correction and evaluation, RobotGPT achieves a 91.5% task success rate—up from 38.5% when using ChatGPT directly—demonstrating significant gains in reliability and real-world deployment potential.

ABSTRACT

We present RobotGPT, an innovative decision framework for robotic manipulation that prioritizes stability and safety. The execution code generated by ChatGPT cannot guarantee the stability and safety of the system. ChatGPT may provide different answers for the same task, leading to unpredictability. This instability prevents the direct integration of ChatGPT into the robot manipulation loop. Although setting the temperature to 0 can generate more consistent outputs, it may cause ChatGPT to lose diversity and creativity. Our objective is to leverage ChatGPT's problem-solving capabilities in robot manipulation and train a reliable agent. The framework includes an effective prompt structure and a robust learning model. Additionally, we introduce a metric for measuring task difficulty to evaluate ChatGPT's performance in robot manipulation. Furthermore, we evaluate RobotGPT in both simulation and real-world environments. Compared to directly using ChatGPT to generate code, our framework significantly improves task success rates, with an average increase from 38.5% to 91.5%. Therefore, training a RobotGPT by utilizing ChatGPT as an expert is a more stable approach compared to directly using ChatGPT as a task planner.

Motivation & Objective

• Address the instability and unpredictability of directly using ChatGPT-generated code in robotic manipulation systems.
• Improve system reliability by training a dedicated agent that learns from ChatGPT’s planning outputs instead of executing them directly.
• Develop an effective prompt structure with self-correction and evaluation modules to enhance task understanding and execution accuracy.
• Introduce a task difficulty metric to analyze the performance boundaries of LLMs in robotic manipulation.
• Demonstrate the framework’s effectiveness in both simulation and real-world environments, including open-ended tasks.

Proposed method

• Design a three-role system: decision bot (generates code), evaluation bot (validates code via a fine-tuned model), and corrector bot (diagnoses and fixes errors).
• Implement a closed-loop execution process where code is tested line-by-line and corrected iteratively based on runtime feedback.
• Use a self-correction mechanism that identifies failure reasons and iteratively improves code until it passes evaluation.
• Train a robust, deployment-ready agent using demonstration data generated from successful ChatGPT-generated code sequences.
• Integrate a task difficulty metric based on execution steps and success rates to assess LLM capabilities across varying task complexities.
• Deploy the trained RobotGPT agent in real-world robotic environments to validate generalization and stability.

Experimental results

Research questions

• RQ1 Can a framework be designed to stabilize and improve the reliability of ChatGPT-generated code for robotic manipulation?
• RQ2 How does the performance of direct ChatGPT code execution compare to a trained agent that learns from such outputs?
• RQ3 What prompt engineering strategies enhance LLM understanding of robotic tasks and environments?
• RQ4 To what extent does task complexity affect the success rate of LLM-based planning in robotics?
• RQ5 Can LLM-driven agents outperform human programmers in open-ended robotic task execution?

Key findings

• RobotGPT achieves a 91.5% task success rate in simulation, a significant improvement from the 38.5% success rate when using ChatGPT directly.
• The framework reduces task failure due to precision errors in placement by enabling iterative code correction and validation.
• In open-ended tasks like room tidying and word spelling, RobotGPT outperformed human participants, generating a 9-letter word ('backfield') compared to the human best of 7 letters.
• RobotGPT demonstrated superior code quality (0.762 vs. 0.70) and reduced time consumption (221.8s vs. 554.9s) in AB testing compared to hand-coding by experienced programmers.
• The task difficulty metric revealed that success rates decrease with increasing task complexity, but RobotGPT maintains high performance across all levels.
• The system successfully deployed and operated in real-world robotic environments, confirming its practical viability beyond simulation.