[Paper Review] Code as Policies: Language Model Programs for Embodied Control

Jacky Liang, Wenlong Huang+6 more|arXiv (Cornell University)|Sep 16, 2022

Robot Manipulation and Learning35 citations

TL;DR

The paper shows that code-writing LLMs can generate robot policy code from natural language commands to control perception-action loops, enabling reactive and waypoint-based policies across multiple robots without additional training, via hierarchical code generation.

ABSTRACT

Large language models (LLMs) trained on code completion have been shown to be capable of synthesizing simple Python programs from docstrings [1]. We find that these code-writing LLMs can be re-purposed to write robot policy code, given natural language commands. Specifically, policy code can express functions or feedback loops that process perception outputs (e.g.,from object detectors [2], [3]) and parameterize control primitive APIs. When provided as input several example language commands (formatted as comments) followed by corresponding policy code (via few-shot prompting), LLMs can take in new commands and autonomously re-compose API calls to generate new policy code respectively. By chaining classic logic structures and referencing third-party libraries (e.g., NumPy, Shapely) to perform arithmetic, LLMs used in this way can write robot policies that (i) exhibit spatial-geometric reasoning, (ii) generalize to new instructions, and (iii) prescribe precise values (e.g., velocities) to ambiguous descriptions ("faster") depending on context (i.e., behavioral commonsense). This paper presents code as policies: a robot-centric formulation of language model generated programs (LMPs) that can represent reactive policies (e.g., impedance controllers), as well as waypoint-based policies (vision-based pick and place, trajectory-based control), demonstrated across multiple real robot platforms. Central to our approach is prompting hierarchical code-gen (recursively defining undefined functions), which can write more complex code and also improves state-of-the-art to solve 39.8% of problems on the HumanEval [1] benchmark. Code and videos are available at https://code-as-policies.github.io

Motivation & Objective

• Motivate grounding language in real-world perception-action loops to reduce data collection and training needs.
• Demonstrate that code-writing LLMs can produce executable robot policies from natural language instructions.
• Propose hierarchical code-generation to compose complex policies and improve generalization.
• Showcase CaP across multiple robots and tasks including reactive control and vision-based manipulation.
• Introduce a robotics-focused code-generation benchmark and analyze scaling effects.

Proposed method

• Use OpenAI Codex to generate Python-like policy code from natural language commands provided as comments.
• Prompt LLMs with examples (few-shot) of instruction-to-code mappings to produce policy code that processes perception outputs and controls actuators.
• Employ hierarchical code-gen by recursively defining undefined functions to build larger, reusable policy modules.
• Execute generated LMPs in real robots by safely running the code via Python exec within a bounded global/local scope.
• Ground LMPs to perception and control APIs (e.g., open-vocabulary detectors, NumPy, Shapely, and PD/impedance-like primitives).
• Evaluate on robotics benchmarks (RoboCodeGen, HumanEval) and real hardware across table-top and mobile manipulation tasks.

Experimental results

Research questions

• RQ1 Can code-writing LLMs translate natural language commands into executable robot policies that reason about perception and control parameters?
• RQ2 Does hierarchical code-generation improve the quality and generalization of robot policies and standard code-gen benchmarks?
• RQ3 How does CaP perform on robotics tasks compared to language-based planners and traditional imitation-learning baselines?
• RQ4 To what extent do perception-grounded open-vocabulary detectors and control primitives enable flexible task grounding for LMPs?
• RQ5 What are the limitations and scaling behavior of CaP across domains and model sizes?

Key findings

• Hierarchical code-generation improves policy quality and code-gen benchmarks, achieving 39.8% P@1 on HumanEval with larger Codex models.
• CaP enables reactive and waypoint-based policies across multiple robot platforms using open-vocabulary perception and programmable control primitives.
• CaP can generalize to unseen instructions and objects, and can adapt policy code to new tasks without additional training.
• On robotics benchmarks, CaP matches or exceeds performance of some supervised baselines in seen-attribute scenarios and shows robust generalization to unseen attributes and tasks.
• Larger models and hierarchical prompting correlate with better performance across code-gen and robotics tasks.