[Paper Review] Improving Planning with Large Language Models: A Modular Agentic Architecture

Taylor W. Webb, Shanka Subhra Mondal+1 more|arXiv (Cornell University)|Sep 30, 2023

Natural Language Processing TechniquesComputer Science3 citations

TL;DR

This paper proposes LLM-PFC, a modular, prefrontal cortex-inspired architecture that improves planning in large language models (LLMs) by decomposing complex tasks into specialized LLM modules—such as task decomposition, action generation, monitoring, prediction, evaluation, and coordination. Evaluated on graph traversal and Tower of Hanoi, the architecture significantly outperforms standard prompting methods, demonstrating that brain-inspired modularity enhances multi-step reasoning in LLMs.

ABSTRACT

Large language models (LLMs) demonstrate impressive performance on a wide variety of tasks, but they often struggle with tasks that require multi-step reasoning or goal-directed planning. Both cognitive neuroscience and reinforcement learning (RL) have proposed a number of interacting functional components that together implement search and evaluation in multi-step decision making. These components include conflict monitoring, state prediction, state evaluation, task decomposition, and orchestration. To improve planning with LLMs, we propose an agentic architecture, the Modular Agentic Planner (MAP), in which planning is accomplished via the recurrent interaction of the specialized modules mentioned above, each implemented using an LLM. MAP improves planning through the interaction of specialized modules that break down a larger problem into multiple brief automated calls to the LLM. We evaluate MAP on three challenging planning tasks -- graph traversal, Tower of Hanoi, and the PlanBench benchmark -- as well as an NLP task requiring multi-step reasoning (strategyQA). We find that MAP yields significant improvements over both standard LLM methods (zero-shot prompting, in-context learning) and competitive baselines (chain-of-thought, multi-agent debate, and tree-of-thought), can be effectively combined with smaller and more cost-efficient LLMs (Llama3-70B), and displays superior transfer across tasks. These results suggest the benefit of a modular and multi-agent approach to planning with LLMs.

Motivation & Objective

• To address the persistent failure of LLMs in multi-step reasoning and goal-directed planning, especially in tasks requiring systematic, hierarchical reasoning.
• To investigate whether structuring LLMs with specialized, functionally distinct modules—inspired by subregions of the human prefrontal cortex (PFC)—can improve planning performance.
• To overcome the limitations of zero-shot prompting and in-context learning by enabling autonomous coordination among specialized LLM components.
• To evaluate whether a black-box, modular architecture based on cognitive neuroscience principles can yield reliable, scalable planning in LLMs.
• To explore whether decomposing planning into discrete, specialized functions (e.g., monitoring, prediction, evaluation) enhances accuracy and reduces hallucination and looping.

Proposed method

• The architecture implements six specialized GPT-4-based modules: TaskDecomposer, Actor, Monitor, Predictor, Evaluator, and TaskCoordinator, each responsible for a distinct cognitive function.
• Each module is prompted with few-shot in-context learning to perform its specific task, such as generating subgoals, proposing actions, or evaluating state transitions.
• The system performs tree search by combining action proposals (Actor), state predictions (Predictor), and state evaluations (Evaluator), enabling backtracking and path selection.
• Error monitoring is enforced via the Monitor module, which rejects actions violating task constraints, reducing hallucination and invalid moves.
• Task coordination is managed by the TaskCoordinator, which tracks subgoal completion and final goal achievement, triggering plan emission when complete.
• The architecture operates as a closed-loop system, with recurrent interactions among modules to iteratively refine and validate plans.

Figure 1: LLM-PFC architecture. The agent receives states from the environment and high-level goals. These are processed by a set of specialized LLM modules. The $\operatorname{TaskDecomposer}$ receives high-level goals and generates a series of subgoals. The $\operatorname{Actor}$ generates propose

Experimental results

Research questions

• RQ1 Can a modular, PFC-inspired architecture improve the planning performance of LLMs on complex, multi-step tasks?
• RQ2 Do specialized LLM modules—each handling a distinct cognitive function—outperform monolithic prompting in reasoning and planning?
• RQ3 To what extent does the coordination of modules like prediction, evaluation, and monitoring reduce hallucination and looping in LLM planning?
• RQ4 Can a black-box, prompting-based modular system achieve better results than standard zero-shot or in-context learning on planning benchmarks?
• RQ5 Does the integration of cognitive neuroscience principles into LLM architecture lead to measurable improvements in systematic reasoning and goal-directed behavior?

Key findings

• The LLM-PFC architecture significantly outperformed standard LLM prompting methods on both graph traversal and Tower of Hanoi planning tasks.
• On the graph traversal task, the architecture achieved high success rates in finding valid paths, reducing hallucinated or invalid transitions common in baseline models.
• In the Tower of Hanoi task, the modular system demonstrated improved success rates over zero-shot prompting, though performance remained suboptimal, suggesting room for refinement.
• The use of a dedicated Monitor module effectively reduced invalid actions and constraint violations, minimizing looping and erroneous state transitions.
• The integration of tree search via Predictor and Evaluator modules enabled systematic exploration of state spaces, enhancing plan quality and reliability.
• The results support the hypothesis that functional specialization and coordinated interaction of LLM modules—modeled after PFC subregions—can enhance planning beyond what is achievable with standard prompting.

Figure 2: Graph Traversal. We investigated two graph traversal tasks utilizing a challenging graph with community structure. Steppath: Find shortest path between two nodes, e.g. node 3 and node 7. Valuepath: Find shortest path from starting location (e.g., node 10) to location with maximum reward (n