[Paper Review] Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy

Zhenyu Guan, Xiangyu Kong+2 more|arXiv (Cornell University)|Jul 9, 2024

Multi-Agent Systems and Negotiation4 citations

TL;DR

This paper introduces Richelieu, a self-evolving LLM-based AI agent for diplomacy that combines strategic planning with memory, social reasoning, and self-play to improve without human data. It outperforms state-of-the-art agents like Cicero in multi-agent Diplomacy by leveraging self-evolved experience and reflection, achieving 44.4% win rate against Cicero in zero-shot evaluations.

ABSTRACT

Diplomacy is one of the most sophisticated activities in human society, involving complex interactions among multiple parties that require skills in social reasoning, negotiation, and long-term strategic planning. Previous AI agents have demonstrated their ability to handle multi-step games and large action spaces in multi-agent tasks. However, diplomacy involves a staggering magnitude of decision spaces, especially considering the negotiation stage required. While recent agents based on large language models (LLMs) have shown potential in various applications, they still struggle with extended planning periods in complex multi-agent settings. Leveraging recent technologies for LLM-based agents, we aim to explore AI's potential to create a human-like agent capable of executing comprehensive multi-agent missions by integrating three fundamental capabilities: 1) strategic planning with memory and reflection; 2) goal-oriented negotiation with social reasoning; and 3) augmenting memory through self-play games for self-evolution without human in the loop.

Motivation & Objective

• To develop an AI diplomacy agent that operates without reliance on domain-specific human demonstration data.
• To enable long-horizon strategic planning and social reasoning in multi-agent environments using LLMs.
• To design a self-evolving framework through self-play games that enhances memory and reflection for continuous improvement.
• To validate the agent’s generalization across different LLM architectures, including GPT-4.0 and Llama 3.
• To demonstrate superior performance in Diplomacy compared to state-of-the-art agents like Cicero.

Proposed method

• The agent employs a hierarchical planning module with memory and reflection to balance short-term actions and long-term goals.
• It uses a social reasoning module to assess trust, detect deception, and adapt to shifting alliances based on partial observations.
• A self-play mechanism generates synthetic experience without human data, enabling continual self-evolution through iterative gameplay.
• The memory module stores and retrieves past negotiations and actions to inform future decisions and support reflective learning.
• The framework integrates LLMs with modular components that can be fine-tuned or replaced, ensuring compatibility with various LLMs like GPT-4.0 and Llama 3.
• The agent evaluates its own decisions through reflection, adjusting strategies based on past outcomes and experience.

Experimental results

Research questions

• RQ1 Can an LLM-based agent achieve human-like diplomatic reasoning in a complex, multi-agent environment without human-annotated negotiation data?
• RQ2 How effective is self-play in generating high-quality experience for training a self-evolving AI diplomacy agent?
• RQ3 To what extent can memory and reflection mechanisms improve long-term strategic planning and social reasoning in multi-agent games?
• RQ4 How does the agent generalize across different LLM architectures, such as GPT-4.0 and Llama 3?
• RQ5 Can the agent outperform state-of-the-art agents like Cicero in Diplomacy without relying on large-scale human demonstrations?

Key findings

• Richelieu achieved a 44.4% win rate against Cicero in zero-shot evaluations, significantly outperforming Cicero’s 94.6% loss rate.
• The agent demonstrated superior negotiation success, with 39.5% win rate when all four alliance conditions were met, compared to Cicero’s 87.5% loss rate.
• Ablation studies confirmed that the combination of memory, reflection, and self-play was essential for performance, with each component contributing to strategic coherence and adaptability.
• The framework generalized well across different LLMs, including GPT-4.0 and Llama 3, showing consistent performance improvements.
• In example cases, Richelieu successfully formed alliances to counter future threats and detected deception, such as England’s false alliance proposal, leading to strategic advantage.
• The agent’s self-evolving mechanism via self-play enabled it to develop complex, long-term strategies without human-annotated data, proving the viability of autonomous agent evolution.