[Paper Review] Modeling Strong and Human-Like Gameplay with KL-Regularized Search

Athul Paul Jacob, David Wu+6 more|arXiv (Cornell University)|Dec 14, 2021

Sports Analytics and Performance4 citations

TL;DR

This paper proposes KL-regularized search to balance strong performance and human-like behavior in multi-agent games. By integrating an imitation-learned policy as a reference via KL divergence regularization in Monte Carlo Tree Search (MCTS) and regret minimization (piKL-hedge), the method achieves superior human prediction accuracy and stronger gameplay than imitation learning alone in chess, Go, Diplomacy, and Hanabi.

ABSTRACT

We consider the task of building strong but human-like policies in multi-agent decision-making problems, given examples of human behavior. Imitation learning is effective at predicting human actions but may not match the strength of expert humans, while self-play learning and search techniques (e.g. AlphaZero) lead to strong performance but may produce policies that are difficult for humans to understand and coordinate with. We show in chess and Go that regularizing search based on the KL divergence from an imitation-learned policy results in higher human prediction accuracy and stronger performance than imitation learning alone. We then introduce a novel regret minimization algorithm that is regularized based on the KL divergence from an imitation-learned policy, and show that using this algorithm for search in no-press Diplomacy yields a policy that matches the human prediction accuracy of imitation learning while being substantially stronger.

Motivation & Objective

• To address the gap between strong AI policies and human-like behavior in multi-agent decision-making.
• To improve human prediction accuracy without sacrificing performance in complex strategic games.
• To develop a unified framework that combines imitation learning with search-based planning for human-aligned strong policies.
• To extend this approach to imperfect-information games where standard MCTS is inapplicable.
• To enable AI agents that are both highly competent and interpretable for human collaboration.

Proposed method

• Uses KL divergence between the search policy and an imitation-learned policy as a regularization term in MCTS to bias search toward human-like behavior.
• Applies the regularization in a novel regret minimization algorithm called piKL-hedge, which incorporates a KL penalty toward a human-imitation anchor policy.
• Modifies the utility function in regret minimization to include a reward term and a KL divergence penalty term with coefficient λ.
• Employs exponential weighting in policy updates based on cumulative regret, with the regularized utility function guiding policy evolution.
• Uses the anchor policy τ as a reference to ensure search policies remain close to human behavior while still optimizing for game performance.
• Applies the method across multiple domains: perfect-information games (chess, Go), single-round imperfect-information games (no-press Diplomacy), and sequential games (Hanabi).

Experimental results

Research questions

• RQ1 Can KL-regularized search improve human prediction accuracy while maintaining or exceeding the strength of purely imitation-learned policies?
• RQ2 How can search-based planning be regularized to produce policies that are both strong and human-like in complex strategic games?
• RQ3 Can regret minimization algorithms be adapted to incorporate KL regularization toward a human-imitation policy in imperfect-information games?
• RQ4 Does KL regularization in search lead to better alignment with human conventions in cooperative or mixed-motive settings?
• RQ5 To what extent does the choice of regularization direction (KL(π||τ) vs. KL(τ||π)) affect convergence and performance in regret minimization?

Key findings

• In chess and Go, MCTS with KL-regularized search using an imitation-learned policy as a prior achieved higher human prediction accuracy than imitation learning alone.
• The method produced policies stronger than those from pure imitation learning while matching or exceeding human prediction accuracy in both chess and Go.
• In no-press Diplomacy, piKL-hedge achieved human prediction accuracy comparable to imitation learning but with significantly stronger head-to-head performance against prior agents.
• The regret minimization algorithm piKL-hedge was proven to converge to an equilibrium where policies are optimal given the anchor policy and deviation costs.
• In Hanabi, applying KL regularization to the SPARTA search algorithm improved human prediction accuracy and greatly enhanced performance over baseline imitation learning.
• The approach consistently produced agents that were both more human-like and stronger than purely imitation-based models across all four domains tested.