[Paper Review] On the Prospects of Incorporating Large Language Models (LLMs) in Automated Planning and Scheduling (APS)

Vishal Pallagani, Kaushik Roy+8 more|arXiv (Cornell University)|Jan 4, 2024

AI-based Problem Solving and Planning129 references7 citations

TL;DR

This position paper surveys 126 studies on using LLMs in APS across eight categories and advocates neuro-symbolic integration with traditional planners.

ABSTRACT

Automated Planning and Scheduling is among the growing areas in Artificial Intelligence (AI) where mention of LLMs has gained popularity. Based on a comprehensive review of 126 papers, this paper investigates eight categories based on the unique applications of LLMs in addressing various aspects of planning problems: language translation, plan generation, model construction, multi-agent planning, interactive planning, heuristics optimization, tool integration, and brain-inspired planning. For each category, we articulate the issues considered and existing gaps. A critical insight resulting from our review is that the true potential of LLMs unfolds when they are integrated with traditional symbolic planners, pointing towards a promising neuro-symbolic approach. This approach effectively combines the generative aspects of LLMs with the precision of classical planning methods. By synthesizing insights from existing literature, we underline the potential of this integration to address complex planning challenges. Our goal is to encourage the ICAPS community to recognize the complementary strengths of LLMs and symbolic planners, advocating for a direction in automated planning that leverages these synergistic capabilities to develop more advanced and intelligent planning systems.

Motivation & Objective

• Motivate integrating LLMs with classical APS to address flexibility and contextual adaptability gaps.
• Categorize LLM applications in APS into eight distinct areas for structured insight.
• Identify gaps and limitations in current LLM-based APS approaches to guide future research.
• Promote a neuro-symbolic AI paradigm as a core direction for advancing APS capabilities.

Proposed method

• Conduct a comprehensive literature review of 126 papers on LLMs in APS.
• Classify papers into eight categories: Language Translation, Plan Generation, Model Construction, Multi-agent Planning, Interactive Planning, Heuristics Optimization, Tool Integration, Brain-Inspired Planning.
• Provide a qualitative synthesis of strengths, gaps, and future opportunities per category.
• Extract and synthesize insights to advocate for neuro-symbolic integration with symbolic planners.
• Use a manual categorization process with cross-author review to ensure coherent taxonomy and coverage.

Figure 1: Radar chart showcasing the relative performance of six language models (GPT-4, Claude-v1, GPT-3.5-turbo, Vicuna-13B, Alpaca-13B, LLama-13B) across key domains: Writing, Roleplay, Reasoning, Math, Coding, Extraction, STEM, and Humanities from Zheng et al. ( 2023a ) .

Experimental results

Research questions

• RQ1 What are the eight categories of LLM applications in APS and what issues/gaps characterize each?
• RQ2 To what extent do LLMs complement or replace traditional symbolic planners across these categories?
• RQ3 What is the evidence for neuro-symbolic integration as a pathway to improved planning systems?

Key findings

• Eight-category taxonomy established for LLMs in APS, with Plan Generation being the most explored (53 papers) and Brain-Inspired Planning least represented (5 papers).
• LLMs excel at Language Translation but face limits in plan optimality, completeness, and generalization in Plan Generation.
• Neuro-symbolic approaches are identified as a promising direction to combine LLMs’ language capabilities with the precision of symbolic planners.
• Interactive Planning and Tool Integration show strong potential for adaptability and cross-system coordination, despite challenges like tool over-reliance and grounding.
• Gaps include grounding, grounding-of-affordances, standardization of inter-agent communication, and the need for world models to improve low-level reasoning.
• The study argues for integrating LLMs with symbolic planners to achieve dynamic, context-aware, and scalable APS.

Figure 2: Of the 126 papers surveyed in this study, 55 were accepted by peer-reviewed conferences. This chart illustrates the distribution of these papers across various conferences in the fields of LLMs and APS, highlighting the primary forums for scholarly contributions in these areas.