[Paper Review] LLM+P: Empowering Large Language Models with Optimal Planning Proficiency

Bo Liu, Yuqian Jiang+5 more|arXiv (Cornell University)|Apr 22, 2023

Topic Modeling84 citations

TL;DR

LLM+P couples large language models with classical planners to translate natural language planning problems into PDDL, solve them optimally with a planner, and translate the plan back to natural language, achieving superior planning performance over plain LLMs.

ABSTRACT

Large language models (LLMs) have demonstrated remarkable zero-shot generalization abilities: state-of-the-art chatbots can provide plausible answers to many common questions that arise in daily life. However, so far, LLMs cannot reliably solve long-horizon planning problems. By contrast, classical planners, once a problem is given in a formatted way, can use efficient search algorithms to quickly identify correct, or even optimal, plans. In an effort to get the best of both worlds, this paper introduces LLM+P, the first framework that incorporates the strengths of classical planners into LLMs. LLM+P takes in a natural language description of a planning problem, then returns a correct (or optimal) plan for solving that problem in natural language. LLM+P does so by first converting the language description into a file written in the planning domain definition language (PDDL), then leveraging classical planners to quickly find a solution, and then translating the found solution back into natural language. Along with LLM+P, we define a diverse set of different benchmark problems taken from common planning scenarios. Via a comprehensive set of experiments on these benchmark problems, we find that LLM+P is able to provide optimal solutions for most problems, while LLMs fail to provide even feasible plans for most problems.\footnote{The code and results are publicly available at https://github.com/Cranial-XIX/llm-pddl.git.

Motivation & Objective

• Motivate the need for reliable long-horizon planning in robotics and demonstrate how classical planners can provide optimal solutions.
• Propose a pipeline (LLM+P) that converts natural language planning problems into PDDL, leverages a classical planner, and translates back to natural language.
• Show that LLM+P outperforms LLM-only planning and LLM-based search methods across diverse robot planning domains.
• Provide a realistic robot demonstration to validate practical applicability in home robotics.

Proposed method

• Use an LLM to translate a natural language planning problem into a PDDL problem file (domain and problem).
• Use a supplied domain PDDL (from a human expert) and feed the generated problem PDDL to a classical planner to obtain a plan, optimally when possible.
• Translate the generated PDDL plan back into natural language for user understanding or action execution.
• Comparison with baselines: LLM-as-P and Tree of Thoughts to assess planning feasibility and optimality.
• Evaluate with seven IPC-inspired domains and 20 tasks per domain, plus a real-robot tidying task demonstration.

Figure 1 : LLM+P makes use of a large language model (LLM) to produce the PDDL description of the given problem, then leverages a classical planner for finding an optimal plan, then translates the raw plan back to natural language using the LLM again.

Experimental results

Research questions

• RQ1 Can state-of-the-art LLMs directly solve long-horizon planning problems (planning feasibility and optimality)?
• RQ2 Does integrating a classical planner via LLM+P improve planning success and optimality across multiple domains?
• RQ3 How crucial is context (example problem and problem PDDL) for LLM+P performance?
• RQ4 Can LLM+P enable a home robot to perform realistic manipulation tasks efficiently from natural language prompts?

Key findings

• LLM-as-P often produces plans in natural language that are not feasible due to lack of precondition reasoning.
• LLM+P yields optimal plans for the majority of problems; most failures are due to mis-specified problem files rather than planner limitations.
• Context (problem-example pair) is crucial for LLM+P to generate correct PDDL problem files.
• LLM+P demonstrates practical effectiveness in a real home-robot tidying task, achieving lower cost than an LLM-only approach.

(a) grasp bottle