[Paper Review] Interactive Task Planning with Language Models

Boyi Li, Philipp Wu+2 more|arXiv (Cornell University)|Oct 16, 2023

Topic Modeling4 citations

TL;DR

This paper proposes ITP (Interactive Task Planning), a training-free framework that uses large language models (LLMs) like GPT-4 to enable real-time, interactive robot task planning and replanning. It combines high-level natural language planning with low-level skill execution via a functional API, achieving 100% success on zero-shot dishwashing tasks and robust adaptation to new goals without prompt engineering or retraining.

ABSTRACT

An interactive robot framework accomplishes long-horizon task planning and can easily generalize to new goals and distinct tasks, even during execution. However, most traditional methods require predefined module design, making it hard to generalize to different goals. Recent large language model based approaches can allow for more open-ended planning but often require heavy prompt engineering or domain specific pretrained models. To tackle this, we propose a simple framework that achieves interactive task planning with language models by incorporating both high-level planning and low-level skill execution through function calling, leveraging pretrained vision models to ground the scene in language. We verify the robustness of our system on the real world task of making milk tea drinks. Our system is able to generate novel high-level instructions for unseen objectives and successfully accomplishes user tasks. Furthermore, when the user sends a new request, our system is able to replan accordingly with precision based on the new request, task guidelines and previously executed steps. Our approach is easy to adapt to different tasks by simply substituting the task guidelines, without the need for additional complex prompt engineering. Please check more details on our https://wuphilipp.github.io/itp_site and https://youtu.be/TrKLuyv26_g.

Motivation & Objective

• To develop a training-free, interactive robotic system that dynamically replans in response to user feedback during task execution.
• To eliminate the need for complex prompt engineering or code-level specifications in task planning for robots.
• To enable generalization to new tasks using only task guidelines and pre-defined low-level skill APIs.
• To demonstrate robust high-level planning and execution in real-world robotic applications such as drink-making and dishwashing.
• To create a flexible, open-source framework that supports rapid adaptation to diverse robotic tasks.

Proposed method

• The system uses GPT-4 as a high-level planner that generates step-by-step instructions based on user requests and task guidelines.
• Low-level robot skills are abstracted into a functional language-based API, enabling direct function calling by the LLM without code-level prompt engineering.
• Completed steps are 'memorized' and used as context during replanning, allowing the system to adapt to new user requests mid-execution.
• The framework relies solely on natural language prompts and task guidelines, avoiding the need for fine-tuning or additional value functions.
• It integrates vision models like Grounded-DINO to detect and localize objects, feeding visual input into the LLM for grounded planning.
• The system dynamically re-plans by combining new user requests, previously executed steps, and task guidelines to generate updated, executable plans.

Experimental results

Research questions

• RQ1 Can a language model-based system generate accurate, executable high-level plans for unseen tasks without fine-tuning or complex prompt engineering?
• RQ2 How well can such a system adapt to mid-execution user requests, incorporating prior completed steps and updated goals?
• RQ3 To what extent can the same framework generalize across distinct robotic tasks using only modified task guidelines and skill APIs?
• RQ4 Can the system maintain plan correctness and task alignment when faced with novel, zero-shot combinations of objects and goals?
• RQ5 What is the impact of using a functional API abstraction for low-level skills on system robustness and ease of deployment?

Key findings

• The system achieved 100% success rate on zero-shot dishwashing tasks, generating correct high-level plans for 27 out of 27 novel requests.
• ITP successfully generated executable plans for novel drink recipes (e.g., taro milk, matcha milk with boba) even when not explicitly listed in the guidelines.
• The system demonstrated robust replanning by accurately adjusting plans in response to new user requests during execution, using memorized completed steps and updated goals.
• Generalization to new tasks required only updating task guidelines and low-level skill definitions, with no changes to the core planning or execution pipeline.
• The framework eliminated the need for code-level prompt engineering or training of value functions, significantly simplifying deployment across new tasks.
• The system maintained plan correctness and alignment with ground truth across diverse, complex task combinations, including multi-object, multi-location dishwashing requests.