[Paper Review] ChatGPT for Robotics: Design Principles and Model Abilities

Sai Vemprala, Rogerio Bonatti+2 more|arXiv (Cornell University)|Feb 20, 2023

Topic Modeling90 citations

TL;DR

The paper studies a perception-action causal transformer (PACT) for autoregressive robotics, examining data collection, tokenization, model size, and sequence length effects on real-time navigation and action prediction. It highlights trade-offs between model capacity, inference speed, and deployment performance.

ABSTRACT

This paper presents an experimental study regarding the use of OpenAI's ChatGPT for robotics applications. We outline a strategy that combines design principles for prompt engineering and the creation of a high-level function library which allows ChatGPT to adapt to different robotics tasks, simulators, and form factors. We focus our evaluations on the effectiveness of different prompt engineering techniques and dialog strategies towards the execution of various types of robotics tasks. We explore ChatGPT's ability to use free-form dialog, parse XML tags, and to synthesize code, in addition to the use of task-specific prompting functions and closed-loop reasoning through dialogues. Our study encompasses a range of tasks within the robotics domain, from basic logical, geometrical, and mathematical reasoning all the way to complex domains such as aerial navigation, manipulation, and embodied agents. We show that ChatGPT can be effective at solving several of such tasks, while allowing users to interact with it primarily via natural language instructions. In addition to these studies, we introduce an open-sourced research tool called PromptCraft, which contains a platform where researchers can collaboratively upload and vote on examples of good prompting schemes for robotics applications, as well as a sample robotics simulator with ChatGPT integration, making it easier for users to get started with using ChatGPT for robotics.

Motivation & Objective

• Understand how a perception-action causal transformer can be pretrained for autonomous robotics tasks.
• Evaluate the impact of dataset size, model depth, and sequence length on navigation accuracy and latency.
• Analyze attention patterns to interpret how temporal dependencies are modeled in robotic control.
• Assess real-time deployment considerations for different model sizes on robotic platforms.

Proposed method

• Collect perception-action data using MuSHR with MPC-driven trajectory libraries in a real office floor plan and in Habitat environments.
• Tokenize sensor modalities with modality-specific backbones (ResNet-18 for RGB, PointNet for 2D LiDAR, and BEV LiDAR with ResNet-18 on real-world data).
• Map discrete actions to embeddings and continuous actions through simple MLPs to produce 128-D tokens.
• Pre-train with a transformer (12 layers, 8 heads, embedding length 128, sequence length 16) using a ramp-up learning rate and weight decay; fine-tune with a smaller learning rate.
• Evaluate model variants (3, 6, 12, 24 layers) on navigation metrics (meters traversed before crash) and action prediction MAE; analyze attention maps and sequence-length effects.

Experimental results

Research questions

• RQ1 How does model size affect pre-training performance and real-time navigation stability in robotic control?
• RQ2 What is the impact of training data size on the PACT model's action prediction and deployment performance?
• RQ3 How do sequence length and attention patterns influence action prediction accuracy and interpretability in robotic tasks?
• RQ4 What are the trade-offs between inference time and accuracy across model variants in real-time robotics deployments?
• RQ5 How do downstream Habitat tasks (mapping/localization) respond to frozen vs. trainable representations derived from PACT?

Key findings

• Increasing training tokens generally improves pre-training performance for the model.
• Larger transformer models do not always improve real-time navigation due to higher inference latency (e.g., 24-layer model is slower than 3-layer model).
• Attention maps show time-ordered dependencies with some heads focusing on starting points or state-change points, varying by layer.
• Longer transformer sequences reduce action-prediction MAE but increase training and deployment latency; a 16-token sequence offers a favorable trade-off.
• Longer sequences yield better action predictions in practice, but deployment latency must be balanced with real-time constraints.
• Habitat downstream tasks visuals indicate results for frozen vs. trainable representations trained from scratch and from PACT.