[論文レビュー] Provable Self-Play Algorithms for Competitive Reinforcement Learning

Yu Bai, Chi Jin|arXiv (Cornell University)|Feb 10, 2020

Reinforcement Learning in Robotics参考文献 34被引用数 30

ひとこと要約

ゼロ和マルコフゲームに対する最初の証明可能なサンプル効率の自己プレイアルゴリズムを提示。VI-ULCBは√T的 regret を実現し、O~(T^{2/3}) の後方探索-探索（explore-then-exploit）変種を含む多項式時間の実装を提供。

ABSTRACT

Self-play, where the algorithm learns by playing against itself without requiring any direct supervision, has become the new weapon in modern Reinforcement Learning (RL) for achieving superhuman performance in practice. However, the majority of exisiting theory in reinforcement learning only applies to the setting where the agent plays against a fixed environment; it remains largely open whether self-play algorithms can be provably effective, especially when it is necessary to manage the exploration/exploitation tradeoff. We study self-play in competitive reinforcement learning under the setting of Markov games, a generalization of Markov decision processes to the two-player case. We introduce a self-play algorithm---Value Iteration with Upper/Lower Confidence Bound (VI-ULCB)---and show that it achieves regret $ ilde{\mathcal{O}}(\sqrt{T})$ after playing $T$ steps of the game, where the regret is measured by the agent's performance against a \emph{fully adversarial} opponent who can exploit the agent's strategy at \emph{any} step. We also introduce an explore-then-exploit style algorithm, which achieves a slightly worse regret of $ ilde{\mathcal{O}}(T^{2/3})$, but is guaranteed to run in polynomial time even in the worst case. To the best of our knowledge, our work presents the first line of provably sample-efficient self-play algorithms for competitive reinforcement learning.

研究の動機と目的

• Motivate and formalize competitive RL under two-player zero-sum Markov games.
• Propose self-play algorithms with provable regret guarantees without restrictive assumptions.
• Analyze computational trade-offs between sample efficiency and runtime.
• Provide PAC-style guarantees and foundational results for self-play in MARL.

提案手法

• Extend UCB concepts to two-player settings by maintaining optimistic and pessimistic Q-estimates (Q^up and Q^low).
• Use Nash equilibrium over a general-sum payoff setup at each state-action pair to select policies (μ, ν) that are jointly greedy w.r.t. (Q^up, Q^low).
• Prove regret upper bounds for VI-ULCB in general MGs and derive a turn-based specialization with improved regret and polynomial runtime.
• Introduce a computationally efficient VI-Explore variant using reward-free exploration to achieve polynomial-time guarantees with O(T^{2/3}) regret.
• Provide PAC-style conversion from regret bounds to sample complexity bounds for near-equilibrium policies.]
• research_questions: ["Can self-play algorithms achieve provable, sample-efficient regret in two-player zero-sum Markov games without restrictive model assumptions?", "How can optimistic/pessimistic value estimates be combined with general-sum Nash equilibria to guide policies in self-play?", "What are the computational and statistical trade-offs between exact regret minimization and polynomial-time approximations in competitive RL?", "Do turn-based Markov games admit improved regret and tractable implementations of self-play algorithms?", "What are the fundamental lower bounds for regret and how close do the proposed methods approach them?"]
• key_findings:["VI-ULCB achieves a regret of Õ(√(H^3 S^2 A B T)) in general zero-sum Markov games (and Õ(√(H^3 S^2 (A+B) T)) in turn-based settings) with high-probability guarantees.", "An explore-then-exploit algorithm achieves Õ(T^{2/3}) regret with guaranteed polynomial runtime.", "A lower bound of Ω(√(H^2 S (A+B) T)) is established for (some) settings, highlighting gaps in S, A, B dependencies and motivating future work.", "The paper provides PAC bounds: with sufficient episodes K, the policy pair derived via VI-ULCB achieves near-equilibrium performance with high probability.", "In turn-based games, the Nash-equilibrium computation reduces to simpler zero-sum subproblems, enabling polynomial-time implementation and improved regret bounds.", "A PPAD-completeness caveat is discussed for the exact Nash_general_sum subroutine, motivating the polynomial-time VI-Explore variant as a practical alternative."]
• table_headers:[

実験結果

主な発見

設定	アルゴリズム	レグレット	PAC	実行時間
General Markov Game	VI-ULCB (Theorem 2)	Õ(√(H^3 S^2 A B T))	Õ(H^4 S^2 A B / ε^2)	PPAD-complete
General Markov Game	VI-explore (Theorem 5)	Õ((H^5 S^2 A B T^2)^{1/3})	Õ(H^5 S^2 A B / ε^2)	Polynomial
Mirror Descent (H=1)	(Rakhlin and Sridharan, 2013)	Õ(√(S(A+B)T))	Õ(S(A+B)/ε^2)	-
Turn-Based Markov Game	VI-ULCB (Corollary 4)	Õ(√(H^3 S^2 (A+B) T))	Õ(H^4 S^2 (A+B)/ε^2)	-
Mirror Descent (H=2)	(Theorem 10)	Õ(√(S(A+B)T))	Õ(S(A+B)/ε^2)	-
Both	Lower Bound (Corollary 7)	Ω(√(H^2 S (A+B) T))	Ω(H^2 S (A+B)/ε^2)	-