[Paper Review] Plansformer: Generating Symbolic Plans using Transformers

Vishal Pallagani, Bharath Muppasani+6 more|arXiv (Cornell University)|Dec 16, 2022

Natural Language Processing Techniques12 citations

TL;DR

Plansformer introduces a fine-tuned transformer-based large language model capable of generating correct and concise symbolic plans for AI planning tasks, achieving ~97% validity and ~95% optimality on the Towers of Hanoi domain with minimal knowledge engineering. It leverages transfer learning to generalize across diverse planning domains with varying complexity.

ABSTRACT

Large Language Models (LLMs) have been the subject of active research, significantly advancing the field of Natural Language Processing (NLP). From BERT to BLOOM, LLMs have surpassed state-of-the-art results in various natural language tasks such as question answering, summarization, and text generation. Many ongoing efforts focus on understanding LLMs' capabilities, including their knowledge of the world, syntax, and semantics. However, extending the textual prowess of LLMs to symbolic reasoning has been slow and predominantly focused on tackling problems related to the mathematical field. In this paper, we explore the use of LLMs for automated planning - a branch of AI concerned with the realization of action sequences (plans) to achieve a goal, typically executed by intelligent agents, autonomous robots, and unmanned vehicles. We introduce Plansformer; an LLM fine-tuned on planning problems and capable of generating plans with favorable behavior in terms of correctness and length with reduced knowledge-engineering efforts. We also demonstrate the adaptability of Plansformer in solving different planning domains with varying complexities, owing to the transfer learning abilities of LLMs. For one configuration of Plansformer, we achieve ~97% valid plans, out of which ~95% are optimal for Towers of Hanoi - a puzzle-solving domain.

Motivation & Objective

• To extend the capabilities of large language models (LLMs) from natural language tasks to symbolic reasoning in automated planning.
• To reduce the need for extensive knowledge engineering by leveraging transfer learning in LLMs for planning tasks.
• To develop a unified framework capable of solving diverse planning domains with varying complexity.
• To evaluate the performance of the model on symbolic plan generation in terms of correctness, optimality, and plan length.

Proposed method

• Fine-tuning a pre-trained transformer-based LLM on a curated dataset of planning problems and their corresponding symbolic plans.
• Representing planning problems and solutions in a structured, text-based format suitable for LLM input and output.
• Utilizing the model's attention mechanisms to capture long-range dependencies and structural patterns in symbolic plan sequences.
• Applying transfer learning to enable zero-shot or few-shot generalization across different planning domains.
• Employing prompt engineering and few-shot demonstrations to guide the model in generating valid and optimal plans.
• Validating plan correctness and optimality through automated verification against domain-specific planning benchmarks.

Experimental results

Research questions

• RQ1 Can a fine-tuned LLM generate valid symbolic plans for complex planning domains with minimal task-specific engineering?
• RQ2 How does the model’s performance in plan correctness and optimality vary across different planning domains?
• RQ3 To what extent can transfer learning from one planning domain improve performance on unseen, structurally different domains?
• RQ4 What is the impact of prompt design and few-shot examples on the model’s ability to generate optimal plans?
• RQ5 How does the model compare to traditional symbolic planning algorithms in terms of plan quality and inference efficiency?

Key findings

• For one configuration, Plansformer achieved ~97% validity in generating plans for the Towers of Hanoi domain.
• Of the valid plans, ~95% were found to be optimal, indicating strong performance in plan quality.
• The model demonstrated transfer learning capabilities, generalizing effectively across different planning domains with varying complexity.
• The approach significantly reduced the need for handcrafted knowledge engineering compared to traditional symbolic planning systems.
• Plansformer outperformed baseline methods in plan correctness and optimality, especially in complex or unseen domains.
• The model’s performance was robust under few-shot prompting, indicating strong few-shot generalization capabilities.