[Paper Review] A Framework for Neurosymbolic Robot Action Planning using Large Language Models

Alessio Capitanelli, Fulvio Mastrogiovanni|arXiv (Cornell University)|Mar 1, 2023

Machine Learning in Materials ScienceMaterials Science28 references3 citations

TL;DR

This paper proposes Teriyaki, a neurosymbolic framework that leverages GPT-3 to generate PDDL-compliant plans action-by-action, enabling concurrent planning and execution. It achieves 95.5% problem-solving accuracy, produces plans up to 13.5% shorter, and reduces average plan waiting time by 61.2% compared to traditional planners.

ABSTRACT

Symbolic task planning is a widely used approach to enforce robot autonomy due to its ease of understanding and deployment in robot architectures. However, techniques for symbolic task planning are difficult to scale in real-world, human-robot collaboration scenarios because of the poor performance in complex planning domains or when frequent re-planning is needed. We present a framework, Teriyaki, specifically aimed at bridging the gap between symbolic task planning and machine learning approaches. The rationale is training Large Language Models (LLMs), namely GPT-3, into a neurosymbolic task planner compatible with the Planning Domain Definition Language (PDDL), and then leveraging its generative capabilities to overcome a number of limitations inherent to symbolic task planners. Potential benefits include (i) a better scalability in so far as the planning domain complexity increases, since LLMs' response time linearly scales with the combined length of the input and the output, and (ii) the ability to synthesize a plan action-by-action instead of end-to-end, making each action available for execution as soon as it is generated instead of waiting for the whole plan to be available, which in turn enables concurrent planning and execution. Recently, significant efforts have been devoted by the research community to evaluate the cognitive capabilities of LLMs, with alternate successes. Instead, with Teriyaki we aim to provide an overall planning performance comparable to traditional planners in specific planning domains, while leveraging LLMs capabilities to build a look-ahead predictive planning model. Preliminary results in selected domains show that our method can: (i) solve 95.5% of problems in a test data set of 1,000 samples; (ii) produce plans up to 13.5% shorter than a traditional symbolic planner; (iii) reduce average overall waiting times for a plan availability by up to 61.4%

Motivation & Objective

• To address the scalability and re-planning inefficiency of symbolic task planners in dynamic, human-robot collaboration environments.
• To enable concurrent planning and execution by generating plans action-by-action instead of end-to-end.
• To leverage the generative capabilities of LLMs to improve planning efficiency and predictability in complex, real-world domains.
• To develop a PDDL-compliant neurosymbolic planner that maintains high accuracy while reducing planning latency.

Proposed method

• Training a GPT-3-based model on synthetic problem pairs generated by a traditional symbolic planner (Probe) to produce PDDL-compliant plans.
• Using a prompt engineering strategy to condition the LLM to output valid PDDL actions and plan structures.
• Implementing a modular architecture where each action is generated sequentially and made available for immediate execution.
• Enabling concurrent planning and execution by decoupling plan generation from full plan completion.
• Leveraging the linear scaling of LLM inference time with input and output length to improve predictability and reduce latency.
• Validating the framework on the MACRO domain using a test set of 1,000 problems to measure accuracy, plan length, and waiting time.

Figure 1 : A Baxter robot executing actions in two domains involving the manipulation of an articulated object. A human can act on an articulated object’s joint at any time, forcing the robot to re-plan.

Experimental results

Research questions

• RQ1 Can a fine-tuned LLM generate PDDL-compliant plans with accuracy comparable to traditional symbolic planners?
• RQ2 Does action-by-action plan generation reduce the average waiting time for plan availability compared to end-to-end planning?
• RQ3 Can the LLM-based approach produce shorter, more efficient plans than a baseline symbolic planner?
• RQ4 How does the predictability of plan generation (measured by standard deviation of response time) compare between LLM-based and traditional planners?

Key findings

• Teriyaki solved 95.5% of the 1,000 test problems, demonstrating accuracy on par with a state-of-the-art symbolic planner.
• The average plan length produced by Teriyaki was up to 13.5% shorter than that of the baseline symbolic planner.
• The average waiting time for a plan to become available was reduced by 61.2% due to action-by-action generation.
• The standard deviation of response time for single actions was reduced from 3.47 to 0.15, indicating significantly improved predictability.
• The framework enables concurrent planning and execution, reducing latency and improving interaction fluency in human-robot collaboration.
• Despite slower raw planning times with GPT-3, the framework's generative and scalable nature offers strong advantages in dynamic, re-planning-intensive scenarios.

Figure 2 : A diagram of Teriyaki fine-tuning process. Blocks in yellow represent custom code developed for data generation and processing as described in section 3.2 .