[Paper Review] Last-iterate Convergence in Extensive-Form Games

Chung‐Wei Lee, Christian Kroer+1 more|arXiv (Cornell University)|Jun 27, 2021

Artificial Intelligence in Games45 references4 citations

TL;DR

This paper establishes last-iterate convergence for optimistic regret-minimization algorithms in two-player zero-sum extensive-form games with perfect recall, demonstrating that algorithms like Dilated Optimistic Multiplicative Weights Update (DOMWU) converge linearly to Nash equilibrium, unlike traditional CFR methods that rely on averaging and converge slowly. The key contribution is a theoretical foundation for faster, direct convergence in sequential games using strongly convex regularizers and optimistic updates.

ABSTRACT

Regret-based algorithms are highly efficient at finding approximate Nash equilibria in sequential games such as poker games. However, most regret-based algorithms, including counterfactual regret minimization (CFR) and its variants, rely on iterate averaging to achieve convergence. Inspired by recent advances on last-iterate convergence of optimistic algorithms in zero-sum normal-form games, we study this phenomenon in sequential games, and provide a comprehensive study of last-iterate convergence for zero-sum extensive-form games with perfect recall (EFGs), using various optimistic regret-minimization algorithms over treeplexes. This includes algorithms using the vanilla entropy or squared Euclidean norm regularizers, as well as their dilated versions which admit more efficient implementation. In contrast to CFR, we show that all of these algorithms enjoy last-iterate convergence, with some of them even converging exponentially fast. We also provide experiments to further support our theoretical results.

Motivation & Objective

• To address the slow convergence and averaging overhead of counterfactual regret minimization (CFR) in extensive-form games.
• To investigate whether optimistic regret-minimization algorithms can achieve last-iterate convergence in zero-sum extensive-form games with perfect recall.
• To establish explicit convergence rates for optimistic mirror descent algorithms using strongly convex regularizers.
• To identify conditions under which last-iterate convergence is provably faster than average-iterate convergence.
• To extend recent advances in last-iterate convergence from normal-form to extensive-form games.

Proposed method

• Formalizing the problem in the sequence-form representation of extensive-form games with perfect recall.
• Applying optimistic online mirror descent (OMD) with strongly convex regularizers, including vanilla entropy, squared Euclidean norm, and their dilated variants.
• Using the dilated entropy regularizer to enable efficient implementation and prove linear convergence under uniqueness of Nash equilibrium.
• Establishing convergence via Bregman divergences and bounding the distance to the optimal strategy using game-dependent constants.
• Analyzing the behavior of optimistic updates to ensure the last iterate approaches the Nash equilibrium, not just the average.
• Proving that algorithms like DOMWU achieve exponential (linear) convergence rates under the assumption of a unique Nash equilibrium.

Experimental results

Research questions

• RQ1 Can optimistic regret-minimization algorithms achieve last-iterate convergence in two-player zero-sum extensive-form games with perfect recall?
• RQ2 Do algorithms like DOMWU converge linearly to Nash equilibrium, and under what conditions?
• RQ3 Why do CFR and its variants fail to converge in the last iterate, despite convergence in the average?
• RQ4 Can the theoretical convergence rate of optimistic algorithms be improved beyond O(1/√T) or O(1/T)?
• RQ5 Is there a theoretical justification for the empirical observation that last iterates of optimistic algorithms outperform averaged strategies in practice?

Key findings

• All optimistic regret-minimization algorithms with strongly convex regularizers, including DOMWU, achieve last-iterate convergence in two-player zero-sum extensive-form games with perfect recall.
• The Dilated Optimistic Multiplicative Weights Update (DOMWU) algorithm converges linearly to the unique Nash equilibrium under the assumption of uniqueness.
• In contrast, standard CFR and its variants like CFR+ do not converge in the last iterate, even empirically, due to their reliance on averaging.
• The convergence rate of DOMWU is proven to be exponential (linear), significantly faster than the O(1/√T) or O(1/T) rates typical of regret-minimization algorithms.
• Theoretical analysis shows that the Bregman divergence to the optimal strategy decays exponentially, which is key to establishing linear convergence.
• While DOGDA shows strong empirical performance and asymptotic linear convergence, a concrete convergence rate remains an open problem due to technical challenges with the dilated Euclidean regularizer.