[Paper Review] PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change

Karthik Valmeekam, Marquez, Matthew+3 more|arXiv (Cornell University)|Jun 21, 2022

Natural Language Processing Techniques49 citations

TL;DR

PlanBench is an extensible benchmark suite that uses IPC-style planning domains (Blocksworld and Logistics) to evaluate LLMs on planning and reasoning about change; results show current LLMs like GPT-4 and InstructGPT-3 struggle on many planning tasks.

ABSTRACT

Generating plans of action, and reasoning about change have long been considered a core competence of intelligent agents. It is thus no surprise that evaluating the planning and reasoning capabilities of large language models (LLMs) has become a hot topic of research. Most claims about LLM planning capabilities are however based on common sense tasks-where it becomes hard to tell whether LLMs are planning or merely retrieving from their vast world knowledge. There is a strong need for systematic and extensible planning benchmarks with sufficient diversity to evaluate whether LLMs have innate planning capabilities. Motivated by this, we propose PlanBench, an extensible benchmark suite based on the kinds of domains used in the automated planning community, especially in the International Planning Competition, to test the capabilities of LLMs in planning or reasoning about actions and change. PlanBench provides sufficient diversity in both the task domains and the specific planning capabilities. Our studies also show that on many critical capabilities-including plan generation-LLM performance falls quite short, even with the SOTA models. PlanBench can thus function as a useful marker of progress of LLMs in planning and reasoning.

Motivation & Objective

• Introduce PlanBench, an extensible benchmark for evaluating LLM planning and reasoning across diverse domains.
• Ground benchmarks in classical planning formalisms (PDDL) to constrain and test LLM outputs.
• Provide a curriculum of tasks (plan generation, cost-optimal planning, verification, execution reasoning, replanning, generalization, reuse, and robustness) to assess planning capabilities.
• Demonstrate a domain-independent assessment framework plus domain-dependent components (domains, problem generation, translation) for prompt generation and plan validation.
• Offer a dataset of prompts (~26,250) including obfuscated domains to test robustness to domain naming.
• Show baseline results with state-of-the-art LLMs to quantify current capabilities and guide future improvements.

Proposed method

• Define a domain-independent planning assessment framework anchored to symbolic planners and plan validators.
• Use domain-dependent components: a lifted domain model, problem generator, and a translator to natural language and back to formal representations.
• Translate PDDL-like problems into natural language prompts and parse LLM outputs back into plan representations for validation.
• Employ a curriculum of eight test cases (seven planning/reasoning tasks plus plan generation) across IPC domains, with few-shot prompts and plan-end markers for extraction.
• Obfuscate domains to test reliance on surface naming versus underlying reasoning patterns.
• Provide a public repository with tools, datasets, and reproduction scripts for prompts and results.

Experimental results

Research questions

• RQ1 Can LLMs generate valid plans that achieve explicit goals in common-sense planning domains?
• RQ2 Are LLMs able to produce cost-optimal plans and verify plan validity under constraints?
• RQ3 Can LLMs reason about the execution of plans and predict outcomes of actions?
• RQ4 How robust are LLMs to goal reformulation, plan reuse, and replanning under changes?
• RQ5 To what extent can LLMs generalize learned planning patterns to new instances or obfuscated domains?

Key findings

• GPT-4 achieves 34.3% correct on Plan Generation (206/600) and 33.0% on Cost-Optimal Planning (198/600).
• InstructGPT-3 attains 6.8% on Plan Generation (41/600) and 5.8% on Cost-Optimal Planning (35/600).
• Plan Verification yields 58.6% correct for GPT-4 (352/600) and 12% for InstructGPT-3 (72/600).
• Reasoning About Plan Execution reaches 31.8% for GPT-4 (191/600) but only 0.6% for InstructGPT-3 (4/600).
• Replanning achieves 48.1% for GPT-4 (289/600) and 6.6% for InstructGPT-3 (40/600).
• Plan Generalization: 28.2% for GPT-4 (141/500) and 9.8% for InstructGPT-3 (49/500).
• Plan Reuse shows 65.3% for GPT-4 (392/600) and 17% for InstructGPT-3 (102/600).
• Robustness to Goal Reformulation (Shuffling): 76.8% for GPT-4 (461/600) and 77.8% for InstructGPT-3 (467/600).
• Robustness to Goal Reformulation (Full→Partial): 87% for GPT-4 (522/600) and 77.8% for InstructGPT-3 (467/600).
• Robustness to Goal Reformulation (Partial→Full): 58% for GPT-4 (348/600) and 60.5% for InstructGPT-3 (363/600).