[Paper Review] Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

Michael J. Ahn, Anthony Brohan+43 more|arXiv (Cornell University)|Apr 4, 2022

Multimodal Machine Learning Applications512 citations

TL;DR

The paper presents SayCan, a framework that grounding large language models (LLMs) in robotics by coupling their high-level planning with learned affordances from pretrained skills, enabling real-world, long-horizon instruction execution on a mobile manipulator.

ABSTRACT

Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. We propose to provide real-world grounding by means of pretrained skills, which are used to constrain the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model's "hands and eyes," while the language model supplies high-level semantic knowledge about the task. We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment. We evaluate our method on a number of real-world robotic tasks, where we show the need for real-world grounding and that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator. The project's website and the video can be found at https://say-can.github.io/.

Motivation & Objective

• Motivate how LLMs lack real-world grounding and can fail when deployed on embodied agents.
• Propose grounding LLM outputs with world-aware affordances from pretrained skills.
• Enable interpretable, step-by-step execution plans that are feasible in the robot’s environment.
• Demonstrate real-world performance on long-horizon kitchen tasks with a mobile robot.

Proposed method

• Represent each low-level skill with a policy and a TD-trained value function (affordance).
• Compute p(ell_pi|i) from the LLM for each skill description ell_pi given the instruction i.
• Compute p(c_pi|s,ell_pi) as the skill’s affordance (success probability from state s).
• Combine scores via p(c_pi|s,ell_pi) * p(ell_pi|i) to select the next skill pi.
• Iteratively execute selected skills and re-query the LLM with updated context until completion.
• Train language-conditioned policies via behavior cloning (BC) or reinforcement learning (RL), with multi-task setups conditioned on text embeddings.

Experimental results

Research questions

• RQ1 Can an embodied agent execute high-level natural language instructions by grounding LLM knowledge in real-world affordances?
• RQ2 Does combining LLM-guided planning with skill affordances improve planning and execution on real robots?
• RQ3 How does the approach scale to long-horizon, abstract tasks in a kitchen environment?
• RQ4 How do different language models and grounding components affect performance?
• RQ5 What is the impact of adding new skills to the system?

Key findings

• PaLM-SayCan achieves 84% planning success and 74% execution success in a mock kitchen.
• In a real kitchen, planning and execution drop to 81% and 60%, respectively, showing reasonable generalization to the real world.
• Affordance grounding plus LLM guidance nearly doubles performance over non-grounded baselines.
• Larger LLMs improve performance; PaLM (540B) outperforms FLAN in planning and execution on the full system.
• Ablations show both language grounding and affordance grounding are necessary for strong performance.
• System can integrate new skills easily (e.g., drawer manipulation) and maintain performance on existing tasks.