[Paper Review] AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents

Petr Anokhin, Nikita Semenov+5 more|arXiv (Cornell University)|Jul 5, 2024

Advanced Graph Neural Networks4 citations

TL;DR

AriGraph proposes a hybrid memory architecture that integrates episodic and semantic memories into a dynamic knowledge graph for LLM agents, enabling structured, scalable, and efficient reasoning in interactive text environments. The Ariadne agent using AriGraph achieves near-human performance in complex text-based games like TextWorld and NetHack, outperforming full history, RAG, and strong RL baselines across multiple tasks.

ABSTRACT

Advancements in the capabilities of Large Language Models (LLMs) have created a promising foundation for developing autonomous agents. With the right tools, these agents could learn to solve tasks in new environments by accumulating and updating their knowledge. Current LLM-based agents process past experiences using a full history of observations, summarization, retrieval augmentation. However, these unstructured memory representations do not facilitate the reasoning and planning essential for complex decision-making. In our study, we introduce AriGraph, a novel method wherein the agent constructs and updates a memory graph that integrates semantic and episodic memories while exploring the environment. We demonstrate that our Ariadne LLM agent, consisting of the proposed memory architecture augmented with planning and decision-making, effectively handles complex tasks within interactive text game environments difficult even for human players. Results show that our approach markedly outperforms other established memory methods and strong RL baselines in a range of problems of varying complexity. Additionally, AriGraph demonstrates competitive performance compared to dedicated knowledge graph-based methods in static multi-hop question-answering.

Motivation & Objective

• To address the limitations of unstructured memory representations in LLM agents, such as full history and retrieval-augmented generation (RAG), which hinder effective reasoning and planning.
• To explore whether integrating episodic and semantic memories into a unified knowledge graph framework improves long-term memory utilization and decision-making in dynamic environments.
• To evaluate whether structured memory representations enhance retrieval efficiency and support effective environmental exploration and task planning.
• To develop a scalable memory architecture that supports complex, multi-step reasoning in interactive text-based environments without relying on pre-existing knowledge.
• To demonstrate that LLM agents can autonomously learn and maintain a coherent world model through interaction, matching or exceeding human-level performance.

Proposed method

• Constructs a knowledge graph where semantic facts (entities and relations) are stored as nodes and edges, forming a structured world model.
• Introduces episodic edges that link events, actions, and observations to specific contexts, embedding temporal and situational details into the graph.
• Uses LLMs to dynamically update the knowledge graph during environment interaction, incorporating new observations and refining existing knowledge.
• Employs a pipeline of memory retrieval, planning, and decision-making, with AriGraph serving as the core memory module for fact retrieval and reasoning.
• Applies graph-based retrieval to efficiently locate relevant facts, enabling faster and more accurate reasoning than vector-based RAG or full history methods.
• Supports both forward and backward reasoning by traversing the graph, allowing the agent to plan actions based on accumulated episodic and semantic knowledge.

Experimental results

Research questions

• RQ1 Can LLM-based agents learn a useful, structured world model from scratch through interaction with an environment?
• RQ2 Does integrating episodic and semantic memories in a unified knowledge graph improve retrieval of relevant facts from memory?
• RQ3 Does the structured representation of memory enhance exploration and decision-making in complex, interactive environments?
• RQ4 How does the AriGraph memory architecture compare to existing memory methods (e.g., full history, RAG, summarization) in terms of performance and scalability?
• RQ5 Can the agent achieve human-level or better performance in complex text-based games using only interaction and self-constructed memory?

Key findings

• The Ariadne agent with AriGraph memory achieved a normalized score of 1.0 on the hardest versions of Treasure Hunt and Cooking tasks in TextWorld, outperforming all baselines and matching top human players.
• In the Cleaning task, AriGraph achieved a normalized score of 0.79, significantly outperforming RAG (0.39), Simulacra (0.7), and full history (0.05).
• On the hardest Cooking task, AriGraph scored 0.65, surpassing Reflexion (0.0), Simulacra (0.0), and RAG (0.17), demonstrating robustness under high complexity.
• Ablation studies show that removing episodic memory reduces performance to 0.67 on the hard Treasure Hunt, confirming its critical role in complex reasoning.
• The AriGraph agent with LLaMA-3-70B achieved a score of 0.47 on Treasure Hunt, indicating that performance scales with model size, but still outperforms other LLM-based memory methods.
• Performance dynamics show that AriGraph maintains high performance across increasing task difficulty, with no degradation in quality, and surpasses human players in speed on the Cooking task.