[Paper Review] KnowAgent: Knowledge-Augmented Planning for LLM-Based Agents

Yuqi Zhu, Shuofei Qiao+8 more|arXiv (Cornell University)|Mar 5, 2024

Semantic Web and Ontologies4 citations

TL;DR

KnowAgent proposes a knowledge-augmented planning framework that enhances LLM-based agents by integrating an external action knowledge base and a knowledgeable self-learning strategy to reduce planning hallucinations. By constraining action trajectories with structured knowledge, it achieves state-of-the-art or comparable performance on HotpotQA and ALFWorld across multiple backbone models.

ABSTRACT

Large Language Models (LLMs) have demonstrated great potential in complex reasoning tasks, yet they fall short when tackling more sophisticated challenges, especially when interacting with environments through generating executable actions. This inadequacy primarily stems from the lack of built-in action knowledge in language agents, which fails to effectively guide the planning trajectories during task solving and results in planning hallucination. To address this issue, we introduce KnowAgent, a novel approach designed to enhance the planning capabilities of LLMs by incorporating explicit action knowledge. Specifically, KnowAgent employs an action knowledge base and a knowledgeable self-learning strategy to constrain the action path during planning, enabling more reasonable trajectory synthesis, and thereby enhancing the planning performance of language agents. Experimental results on HotpotQA and ALFWorld based on various backbone models demonstrate that KnowAgent can achieve comparable or superior performance to existing baselines. Further analysis indicates the effectiveness of KnowAgent in terms of planning hallucinations mitigation. Code is available in https://github.com/zjunlp/KnowAgent.

Motivation & Objective

• To address planning hallucinations in LLM-based agents caused by insufficient action knowledge during trajectory generation.
• To improve the reasoning and planning capabilities of language agents in complex, interactive environments.
• To reduce reliance on human-annotated trajectories by enabling self-learning from model-generated plans.
• To demonstrate the effectiveness of external action knowledge in guiding plausible and executable action sequences.
• To enable broader deployment of LLM agents in real-world applications such as question-answering, web browsing, and robotics.

Proposed method

• Construction of a structured action knowledge base containing task-specific action rules and dependencies for environments like ALFWorld and HotpotQA.
• Conversion of action knowledge into natural language prompts to enable LLMs to reason over and apply the knowledge during planning.
• Integration of the action knowledge base into the agent’s planning process via a prompt-based framework that guides Thought-Action-Observation (TAO) trajectory generation.
• Implementation of a knowledgeable self-learning phase where the agent iteratively improves its planning by refining trajectories using feedback and knowledge constraints.
• Use of a TAO (Thought-Action-Observation) trajectory format to model agent interactions, with history-aware generation conditioned on prior steps.
• Application of knowledge-aware prompting to ensure action sequences follow logical and physical constraints, such as requiring 'go to receptacle' before 'open receptacle'.

Experimental results

Research questions

• RQ1 Can external action knowledge reduce planning hallucinations in LLM-based agents?
• RQ2 How does incorporating structured action knowledge improve planning performance on complex reasoning tasks?
• RQ3 To what extent can self-learning from model-generated trajectories improve planning accuracy without human-annotated data?
• RQ4 Does the knowledge-augmented approach generalize across different backbone LLMs and environments?
• RQ5 Can manually refined action knowledge from LLMs reduce human labor while maintaining or improving performance?

Key findings

• KnowAgent significantly reduces planning hallucinations by enforcing logical action dependencies, such as requiring 'go to' before 'open' or 'take'.
• On the HotpotQA benchmark, KnowAgent achieves performance comparable to or better than existing baselines across multiple LLM backbones.
• On the ALFWorld benchmark, KnowAgent demonstrates superior or competitive performance on tasks like 'Clean', 'Heat', 'Cool', and 'Pick Two'.
• The knowledgeable self-learning strategy improves planning quality over iterations, showing that model-generated trajectories can be refined effectively with knowledge constraints.
• The use of LLM-generated, manually refined action knowledge reduces human annotation effort while maintaining high performance.
• Ablation studies confirm that both the action knowledge base and the self-learning mechanism are essential for mitigating hallucinations and improving planning outcomes.