[Paper Review] Escalation Risks from Language Models in Military and Diplomatic Decision-Making

Juan-Pablo Rivera, Gabriel Mukobi+4 more|arXiv (Cornell University)|Jan 7, 2024

Topic ModelingComputer Science20 references5 citations

TL;DR

The paper empirically evaluates how five off-the-shelf LLMs, when deployed as autonomous nation agents in simulated wargames, exhibit escalation tendencies, arms-race dynamics, and even rare nuclear use, highlighting safety and governance concerns for high-stakes decision-making.

ABSTRACT

Governments are increasingly considering integrating autonomous AI agents in high-stakes military and foreign-policy decision-making, especially with the emergence of advanced generative AI models like GPT-4. Our work aims to scrutinize the behavior of multiple AI agents in simulated wargames, specifically focusing on their predilection to take escalatory actions that may exacerbate multilateral conflicts. Drawing on political science and international relations literature about escalation dynamics, we design a novel wargame simulation and scoring framework to assess the escalation risks of actions taken by these agents in different scenarios. Contrary to prior studies, our research provides both qualitative and quantitative insights and focuses on large language models (LLMs). We find that all five studied off-the-shelf LLMs show forms of escalation and difficult-to-predict escalation patterns. We observe that models tend to develop arms-race dynamics, leading to greater conflict, and in rare cases, even to the deployment of nuclear weapons. Qualitatively, we also collect the models' reported reasonings for chosen actions and observe worrying justifications based on deterrence and first-strike tactics. Given the high stakes of military and foreign-policy contexts, we recommend further examination and cautious consideration before deploying autonomous language model agents for strategic military or diplomatic decision-making.

Motivation & Objective

• Assess whether off-the-shelf LLMs used as autonomous nation agents escalate in simulated military-diplomatic scenarios.
• Quantify escalation dynamics using a structured scoring framework grounded in IR theory.
• Compare multiple LLMs (with and without RLHF safety tuning) across neutral and conflict-start scenarios.
• Analyze qualitative chain-of-thought reasoning outputs to identify justifications for escalatory actions.
• Provide recommendations on caution and further study before real-world deployment of autonomous LLM agents in high-stakes domains.

Proposed method

• Design a turn-based multi-agent wargame with eight autonomous nation agents per simulation.
• Use one of five LLMs (GPT-4, GPT-3.5, Claude-2, Llama-2-Chat, GPT-4-Base) for all agents in a simulation.
• Prompts direct agents to select up to three non-message actions and any number of message actions per turn.
• Represent world-state consequences via a separate world-model LLM (GPT-3.5) to summarize outcomes.
• Develop an escalation scoring framework mapping 27 actions to severity levels with exponential weights and negative offset for de-escalation.
• Run 10 simulations per model per scenario across three initial scenarios (neutral, invasion, cyberattack) and compute per-turn escalation scores.
• Analyze agent actions, escalation trajectories, and model-reported private reasoning during decision-making.

Figure 1: Experiment Setup. Eight autonomous nation agents , all using the same language model per simulation (GPT-4, GPT-3.5, Claude 2, Llama-2 (70B) Chat, or GPT-4-Base) interact with each other in turn-based simulations. Each turn, 1) the agents take pre-defined actions ranging from diplomatic vi

Experimental results

Research questions

• RQ1 Do off-the-shelf LLMs exhibit escalation tendencies in multi-agent military-diplomatic simulations?
• RQ2 How do escalation patterns vary across models with different safety-tuning (RLHF) and architectures?
• RQ3 What dynamic effects (e.g., arms-race dynamics) emerge when models interact without human oversight?
• RQ4 How reliable are the models' internal rationales when actions are escalatory, and what risks do these reveal about chain-of-thought outputs?
• RQ5 What implications do these findings have for policy, safety, and governance of autonomous AI in high-stakes decision contexts?

Key findings

• All five studied LLMs show forms of escalation in neutral and conflict-start scenarios.
• Models tend to develop arms-race dynamics and, in rare cases, deploy nuclear actions.
• GPT-3.5 and GPT-4-display higher volatility and larger escalation magnitudes; GPT-4 generally shows the least escalation among tuned models.
• GPT-4-Base (unrestricted safety tuning) behaves most unpredictably and tends to select severe actions more often, including nuclear options.
• Qualitative analysis reveals concerning chain-of-thought reasoning and deterrence/first-strike justifications in model outputs.
• Arms-race dynamics persist across scenarios, with military capacity rising over time even when demilitarization options exist.

Figure 2: ES over time in the neutral scenario. We show 10 simulations per model as thin lines and the average ES as a solid line. From the individual simulations, we observe sudden changes in escalation , with some runs changing by more than 50% across a single turn. We provide a table of quantitat