[Paper Review] Text2Motion: From Natural Language Instructions to Feasible Plans

Kevin Lin, Christopher Agia+3 more|arXiv (Cornell University)|Mar 21, 2023

AI-based Problem Solving and Planning71 references10 citations

TL;DR

Text2Motion couples large language model planning with a library of learned skills and a geometric feasibility planner to produce and verify multi-step, long-horizon manipulation plans. It achieves an 82% success rate on challenging tasks, outperforming prior language-based planners at 13%.

ABSTRACT

We propose Text2Motion, a language-based planning framework enabling robots to solve sequential manipulation tasks that require long-horizon reasoning. Given a natural language instruction, our framework constructs both a task- and motion-level plan that is verified to reach inferred symbolic goals. Text2Motion uses feasibility heuristics encoded in Q-functions of a library of skills to guide task planning with Large Language Models. Whereas previous language-based planners only consider the feasibility of individual skills, Text2Motion actively resolves geometric dependencies spanning skill sequences by performing geometric feasibility planning during its search. We evaluate our method on a suite of problems that require long-horizon reasoning, interpretation of abstract goals, and handling of partial affordance perception. Our experiments show that Text2Motion can solve these challenging problems with a success rate of 82%, while prior state-of-the-art language-based planning methods only achieve 13%. Text2Motion thus provides promising generalization characteristics to semantically diverse sequential manipulation tasks with geometric dependencies between skills.

Motivation & Objective

• Motivate long-horizon robot planning where natural language instructions must be converted into feasible symbolic and geometric plans.
• Integrate an LLM with a library of manipulation skills and a geometric feasibility planner to verify plan feasibility before execution.
• Develop a hybrid planning strategy that combines shooting-based and search-based planning to handle unseen tasks.
• Provide a plan-termination mechanism that infers goal states from natural language instructions to verify completion before execution.

Proposed method

• Use an LLM to generate goal propositions and candidate skill sequences from natural language instructions and scene descriptions.
• Represent each skill as a policy with a parameterized manipulation primitive and corresponding Q-function to evaluate feasibility.
• Apply geometric feasibility planning (STAP) to maximize the product of per-skill success probabilities along a plan (Eq. 4–5).
• Implement a shooting-based planner that generates K candidate full skill sequences and selects the best by feasibility scores (Algorithm 1).
• Implement a greedy-search planner that iteratively selects the next skill by combining LLM usefulness with geometric feasibility (Eq. 8–12) and then interleaves shooting when possible.
• Propose a hybrid Text2Motion algorithm that alternates shooting and greedy-step planning to find geometrically feasible plans before execution (Algorithm 3).
• Incorporate an out-of-distribution detector based on Q-value ensemble variance to reject invalid OOD skills (Eq. 13).

Figure 1: To carry out the instruction “get two primary-colored objects onto the rack,” the robot must apply symbolic reasoning over the scene description and language instruction to deduce what skills should be executed to acquire a second primary-colored object, after noticing that a red object is

Experimental results

Research questions

• RQ1 How can we verify the correctness and feasibility of LLM-generated plans for long-horizon robotic manipulation?
• RQ2 Does integrating geometric feasibility planning with LLMs improve success on tasks with geometric dependencies?
• RQ3 How does a hybrid shooting-and-search planning strategy compare to myopic language-based planners in terms of robustness to partial affordance perception?
• RQ4 Can a priori goal prediction reliably terminate planning before execution?
• RQ5 What are the benefits and limitations of grounding plans in a skill library with learned dynamics and Q-functions?

Key findings

• Text2Motion achieves an 82% success rate on a suite of long-horizon table-top manipulation tasks.
• Prior state-of-the-art language-based planning methods achieve about 13% under the same evaluation.
• Geometric feasibility planning across a skill sequence is essential to handle dependencies spanning multiple steps.
• A hybrid planner combining shooting and greedy-search outperforms purely myopic or purely planning-based baselines in tasks with geometric dependencies.
• A priori goal prediction provides a reliable termination signal for planning before execution.

Figure 2: shooting and greedy-search planning overview . Both shooting and greedy-search planners use the LLM to predict the set of valid goal states given the user’s natural language instruction and a description of the current state of the environment. These predicted goals are used to decide when