[Paper Review] Task and Motion Planning with Large Language Models for Object Rearrangement

Yan Ding, Xiaohan Zhang+2 more|arXiv (Cornell University)|Mar 10, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes LLM-GROP, a framework that leverages large language models (LLMs) to extract commonsense spatial knowledge for task and motion planning in multi-object rearrangement. By prompting LLMs to generate semantically valid object configurations and grounding them via motion planning, LLM-GROP enables mobile manipulators to execute natural-language commands in complex, unseen environments with high success rates and human-aligned arrangements.

ABSTRACT

Multi-object rearrangement is a crucial skill for service robots, and commonsense reasoning is frequently needed in this process. However, achieving commonsense arrangements requires knowledge about objects, which is hard to transfer to robots. Large language models (LLMs) are one potential source of this knowledge, but they do not naively capture information about plausible physical arrangements of the world. We propose LLM-GROP, which uses prompting to extract commonsense knowledge about semantically valid object configurations from an LLM and instantiates them with a task and motion planner in order to generalize to varying scene geometry. LLM-GROP allows us to go from natural-language commands to human-aligned object rearrangement in varied environments. Based on human evaluations, our approach achieves the highest rating while outperforming competitive baselines in terms of success rate while maintaining comparable cumulative action costs. Finally, we demonstrate a practical implementation of LLM-GROP on a mobile manipulator in real-world scenarios. Supplementary materials are available at: https://sites.google.com/view/llm-grop

Motivation & Objective

• To address the challenge of enabling service robots to perform common-sense object rearrangement from underspecified natural language instructions.
• To bridge the gap between high-level semantic knowledge (e.g., fork on left of knife) and low-level motion planning in dynamic, unknown environments.
• To improve robot planning generalization across varying scene geometries and object configurations without requiring task-specific training data.
• To evaluate whether LLM-extracted commonsense knowledge leads to more human-preferred and efficient rearrangement plans compared to existing baselines.

Proposed method

• Use a large language model (LLM) to generate symbolic spatial relationships (e.g., 'fork on left of knife') based on natural language commands.
• Ground the symbolic spatial relationships into geometric configurations using a task and motion planner that evaluates physical feasibility.
• Optimize the resulting task-motion plans by balancing motion feasibility, task completion efficiency, and long-term utility.
• Integrate the LLM-generated plans into a real-world mobile manipulation system using a Segway-based platform with a UR5e arm.
• Apply hard-coded grasping policies and obstacle avoidance to enable real-world deployment on a physical robot.
• Use human evaluation to assess user preference for final object arrangements, comparing LLM-GROP against baselines in terms of satisfaction and execution time.

Figure 1: A mobile manipulator is assigned the task of setting a table in a dining domain. The manipulator needs to arrange several tableware objects, including a knife, a fork, a plate, a cup mat, and a mug. These objects are available on the other tables, and there are also randomly generated obst

Experimental results

Research questions

• RQ1 Can LLMs reliably extract commonsense spatial relationships for object rearrangement from natural language instructions?
• RQ2 How does LLM-GROP’s performance in human-aligned object arrangement compare to existing task and motion planning baselines?
• RQ3 To what extent does LLM-GROP generalize across varying scene geometries and object counts without retraining?
• RQ4 Can LLM-extracted knowledge lead to more efficient and preferred arrangements in real-world robotic execution?
• RQ5 What is the trade-off between planning efficiency and human preference in LLM-GROP compared to baselines?

Key findings

• LLM-GROP achieved the highest user rating in human evaluations, significantly outperforming baselines in perceived quality of object arrangements.
• The approach demonstrated the shortest average task completion time among all methods, indicating high efficiency in real-world execution.
• LLM-GROP maintained comparable cumulative action costs to baselines while achieving superior user satisfaction, indicating a favorable trade-off between efficiency and preference.
• The framework successfully executed complex rearrangement tasks on a real mobile manipulator, including placing a fork on the left of a plate and a strawberry on a plate, with semantically valid configurations.
• User ratings remained consistently high across all tasks, including those with five objects, demonstrating robustness to increasing task complexity.
• The real-world demonstration confirmed that LLM-GROP enables robots to avoid obstacles and humans while achieving plausible, human-aligned object placements in dynamic indoor environments.

Figure 2: LLM-GROP takes service requests from humans for setting tables and produces a task-motion plan that the robot can execute. LLM-GROP is comprised of two key components: the LLM and the Task and Motion Planner. The LLM is responsible for creating both symbolic and geometric spatial relations