[Paper Review] Online Double Oracle

Le Cong Dinh, Yaodong Yang+7 more|arXiv (Cornell University)|Mar 13, 2021

Advanced Bandit Algorithms Research3 references4 citations

TL;DR

This paper proposes Online Double Oracle (ODO), a novel algorithm that combines online learning with Double Oracle methods to solve large-scale two-player zero-sum normal-form games. By leveraging no-regret learning and dynamic strategy pool expansion, ODO achieves a regret bound of $\mathcal{O}(\sqrt{Tk\log(k)})$, where $k$ depends on the Nash equilibrium support size rather than the full action space, enabling efficient convergence and strategic exploitation of adversaries—outperforming DO, PSRO, and MWU in convergence rate and payoff across real-world games.

ABSTRACT

Solving strategic games with huge action space is a critical yet under-explored topic in economics, operations research and artificial intelligence. This paper proposes new learning algorithms for solving two-player zero-sum normal-form games where the number of pure strategies is prohibitively large. Specifically, we combine no-regret analysis from online learning with Double Oracle (DO) methods from game theory. Our method -- \emph{Online Double Oracle (ODO)} -- is provably convergent to a Nash equilibrium (NE). Most importantly, unlike normal DO methods, ODO is \emph{rationale} in the sense that each agent in ODO can exploit strategic adversary with a regret bound of $\mathcal{O}(\sqrt{T k \log(k)})$ where $k$ is not the total number of pure strategies, but rather the size of \emph{effective strategy set} that is linearly dependent on the support size of the NE. On tens of different real-world games, ODO outperforms DO, PSRO methods, and no-regret algorithms such as Multiplicative Weight Update by a significant margin, both in terms of convergence rate to a NE and average payoff against strategic adversaries.

Motivation & Objective

• To address the challenge of solving two-player zero-sum normal-form games with prohibitively large action spaces.
• To develop a scalable algorithm that maintains rationality (no-regret) while avoiding dependence on the full game size.
• To enable strategic exploitation of adversaries during gameplay, unlike traditional Double Oracle methods.
• To achieve faster convergence to Nash equilibrium and higher average payoff against strategic opponents.
• To reduce reliance on full game matrix knowledge and minimize expensive best-response oracle calls.

Proposed method

• ODO integrates no-regret online learning with the Double Oracle framework, using regret-minimizing strategies instead of fixed best-response oracles.
• At each iteration, the algorithm computes an $\epsilon$-best response using a strategy pool, then adds it to the player's active strategy set.
• The regret bound is derived via online learning analysis, scaling with $k$, the support size of the Nash equilibrium, not $n$, the total number of pure strategies.
• ODO uses a dynamic strategy pool that grows incrementally, ensuring convergence to a Nash equilibrium under mild assumptions.
• The method avoids full game matrix access, relying only on oracle queries for best responses in sub-games.
• It supports both self-play and opponent-exploitation settings, with performance evaluated via exploitability and average payoff metrics.

Experimental results

Research questions

• RQ1 Can a Double Oracle-style algorithm achieve no-regret learning in large-action zero-sum games?
• RQ2 Does the regret bound of ODO scale with the support size of the Nash equilibrium rather than the full action space?
• RQ3 Can ODO outperform standard no-regret algorithms like MWU and DO in convergence speed and payoff against strategic opponents?
• RQ4 How does ODO perform in real-world games with complex strategy structures, such as poker variants?
• RQ5 Can ODO actively exploit irrational or restricted opponents, unlike conservative DO or PSRO methods?

Key findings

• ODO achieves a regret bound of $\mathcal{O}(\sqrt{Tk\log(k)})$, where $k$ is the support size of the Nash equilibrium, not the total number of pure strategies.
• In 15 real-world zero-sum games, ODO converges to a Nash equilibrium significantly faster than MWU, FP, DO, and PSRO.
• Against an MWU adversary, ODO achieves higher average payoff than MWU and PSRO, demonstrating active exploitation capability.
• On Leduc and Kuhn Poker, ODO achieves lower exploitability than all DO and PSRO baselines and nearly matches CFR, using fewer best-response oracle calls.
• In restricted-opponent settings, ODO quickly attains positive expected payoff, while PSRO remains near constant, confirming its exploitative advantage.
• The number of best-response oracle calls in ODO is substantially lower than in PSRO and comparable to state-of-the-art solvers like CFR, despite superior convergence.