[Paper Review] Provable Self-Play Algorithms for Competitive Reinforcement Learning

Yu Bai, Chi Jin|arXiv (Cornell University)|Feb 10, 2020

Reinforcement Learning in Robotics34 references30 citations

TL;DR

Presents the first provably sample-efficient self-play algorithms for zero-sum Markov games, including VI-ULCB with O~(√T) regret and a polynomial-time explore-then-exploit variant with O~(T^{2/3}) regret.

ABSTRACT

Self-play, where the algorithm learns by playing against itself without requiring any direct supervision, has become the new weapon in modern Reinforcement Learning (RL) for achieving superhuman performance in practice. However, the majority of exisiting theory in reinforcement learning only applies to the setting where the agent plays against a fixed environment; it remains largely open whether self-play algorithms can be provably effective, especially when it is necessary to manage the exploration/exploitation tradeoff. We study self-play in competitive reinforcement learning under the setting of Markov games, a generalization of Markov decision processes to the two-player case. We introduce a self-play algorithm---Value Iteration with Upper/Lower Confidence Bound (VI-ULCB)---and show that it achieves regret $ ilde{\mathcal{O}}(\sqrt{T})$ after playing $T$ steps of the game, where the regret is measured by the agent's performance against a \emph{fully adversarial} opponent who can exploit the agent's strategy at \emph{any} step. We also introduce an explore-then-exploit style algorithm, which achieves a slightly worse regret of $ ilde{\mathcal{O}}(T^{2/3})$, but is guaranteed to run in polynomial time even in the worst case. To the best of our knowledge, our work presents the first line of provably sample-efficient self-play algorithms for competitive reinforcement learning.

Motivation & Objective

• Motivate and formalize competitive RL under two-player zero-sum Markov games.
• Propose self-play algorithms with provable regret guarantees without restrictive assumptions.
• Analyze computational trade-offs between sample efficiency and runtime.
• Provide PAC-style guarantees and foundational results for self-play in MARL.

Proposed method

• Extend UCB concepts to two-player settings by maintaining optimistic and pessimistic Q-estimates (Q^up and Q^low).
• Use Nash equilibrium over a general-sum payoff setup at each state-action pair to select policies (μ, ν) that are jointly greedy w.r.t. (Q^up, Q^low).
• Prove regret upper bounds for VI-ULCB in general MGs and derive a turn-based specialization with improved regret and polynomial runtime.
• Introduce a computationally efficient VI-Explore variant using reward-free exploration to achieve polynomial-time guarantees with O(T^{2/3}) regret.
• Provide PAC-style conversion from regret bounds to sample complexity bounds for near-equilibrium policies.

Experimental results

Research questions

• RQ1 Can self-play algorithms achieve provable, sample-efficient regret in two-player zero-sum Markov games without restrictive model assumptions?
• RQ2 How can optimistic/pessimistic value estimates be combined with general-sum Nash equilibria to guide policies in self-play?
• RQ3 What are the computational and statistical trade-offs between exact regret minimization and polynomial-time approximations in competitive RL?
• RQ4 Do turn-based Markov games admit improved regret and tractable implementations of self-play algorithms?
• RQ5 What are the fundamental lower bounds for regret and how close do the proposed methods approach them?

Key findings

• VI-ULCB achieves a regret of Õ(√(H^3 S^2 A B T)) in general zero-sum Markov games (and Õ(√(H^3 S^2 (A+B) T)) in turn-based settings) with high-probability guarantees.
• An explore-then-exploit algorithm achieves Õ(T^{2/3}) regret with guaranteed polynomial runtime.
• A lower bound of Ω(√(H^2 S (A+B) T)) is established for (some) settings, highlighting gaps in S, A, B dependencies and motivating future work.
• The paper provides PAC bounds: with sufficient episodes K, the policy pair derived via VI-ULCB achieves near-equilibrium performance with high probability.
• In turn-based games, the Nash-equilibrium computation reduces to simpler zero-sum subproblems, enabling polynomial-time implementation and improved regret bounds.
• A PPAD-completeness caveat is discussed for the exact Nash_general_sum subroutine, motivating the polynomial-time VI-Explore variant as a practical alternative.