[Paper Review] Complexity and Algorithms for Exploiting Quantal Opponents in Large Two-Player Games

David Milec, Jakub Černý+2 more|arXiv (Cornell University)|Sep 30, 2020

Game Theory and Applications29 references4 citations

TL;DR

This paper proposes a scalable CFR-based algorithm, RQR (Restricted Quantal Response), to compute strategies that outperform both Nash and Quantal Nash equilibria when facing quantal response (QR) opponents in large two-player games. The method improves exploitability and expected utility against QR and rational opponents, demonstrating superior performance in normal-form and extensive-form games, including large-scale poker variants and Goofspiel 7.

ABSTRACT

Solution concepts of traditional game theory assume entirely rational players; therefore, their ability to exploit subrational opponents is limited. One type of subrationality that describes human behavior well is the quantal response. While there exist algorithms for computing solutions against quantal opponents, they either do not scale or may provide strategies that are even worse than the entirely-rational Nash strategies. This paper aims to analyze and propose scalable algorithms for computing effective and robust strategies against a quantal opponent in normal-form and extensive-form games. Our contributions are: (1) we define two different solution concepts related to exploiting quantal opponents and analyze their properties; (2) we prove that computing these solutions is computationally hard; (3) therefore, we evaluate several heuristic approximations based on scalable counterfactual regret minimization (CFR); and (4) we identify a CFR variant that exploits the bounded opponents better than the previously used variants while being less exploitable by the worst-case perfectly-rational opponent.

Motivation & Objective

• To address the challenge of computing effective strategies against bounded-rational (quantal response) opponents in large two-player games.
• To identify a scalable alternative to computationally hard Quantal Stackelberg Equilibrium (QSE) that maintains high utility and low exploitability.
• To evaluate and improve upon existing regret-minimization heuristics, particularly CFR variants, for exploiting QR opponents.
• To demonstrate that QNE is a poor approximation of QSE and may underperform even standard Nash equilibria in exploitative settings.
• To develop and validate a practical, scalable algorithm that outperforms baselines in both utility and robustness across varying levels of opponent rationality.

Proposed method

• Proposes two solution concepts: Quantal Nash Equilibrium (QNE) and Quantal Stackelberg Equilibrium (QSE), analyzing their theoretical properties and computational hardness.
• Proves that computing QNE is PPAD-hard in normal-form games and QSE is NP-hard in extensive-form games, even in zero-sum settings.
• Employs Counterfactual Regret Minimization (CFR) with a novel variant, RQR, which restricts the opponent's response to better align with quantal response behavior.
• Uses a restricted response mechanism in CFR to simulate a Stackelberg commitment, improving exploitability without sacrificing robustness.
• Employs heuristic approximations based on CFR-f and evaluates performance using expected utility and exploitability metrics across diverse game classes.
• Employs a rationality parameter λ to model varying degrees of opponent bounded rationality and evaluates strategies across different λ values.

Experimental results

Research questions

• RQ1 Can scalable regret minimization algorithms effectively exploit quantal response opponents in large two-player games?
• RQ2 How does the performance of QNE compare to QSE and Nash equilibrium in terms of utility and exploitability?
• RQ3 Is there a CFR-based heuristic that outperforms QNE in both exploitability and expected utility against QR opponents?
• RQ4 Can the proposed method maintain robustness when the opponent is perfectly rational, despite being designed for bounded rationality?
• RQ5 How does the RQR algorithm scale and perform on large games such as Goofspiel 7 and Leduc Hold’em?

Key findings

• RQR achieves a gain of 2.412 in Leduc Hold’em against a CLQR opponent, outperforming QNE (2.357) and CFR (1.191), with lower exploitability (3.849 vs. 4.045).
• In Goofspiel 7, RQR achieves a 2.412 gain with exploitability 3.849, significantly outperforming CFR-QR (gain 2.357, exploitability 4.045) and CFR (gain 1.191, exploitability 0.115).
• RQR consistently outperforms QNE in both expected utility and exploitability across varying rationality parameters λ, even when λ is high (i.e., opponent is nearly rational).
• GA and COMB are infeasible for large games like Goofspiel 7 due to memory and computational constraints, while RQR scales effectively with 1000 iterations.
• The RQR variant is less exploitable than QNE and performs better than NE and QNE across all tested games and rationality levels.
• The study confirms that QNE is a poor proxy for QSE and may yield worse performance than standard Nash strategies in exploitative settings.