[论文解读] Provable Self-Play Algorithms for Competitive Reinforcement Learning

Yu Bai, Chi Jin|arXiv (Cornell University)|Feb 10, 2020

Reinforcement Learning in Robotics参考文献 34被引用 30

一句话总结

呈现首个可被证明样本效率自我博弈算法，用于零和马尔可夫博弈，包括 O~(√T) 遗憾的 VI-ULCB，以及一个多项式时间的探索-再利用变体，遗憾为 O~(T^{2/3}).

ABSTRACT

Self-play, where the algorithm learns by playing against itself without requiring any direct supervision, has become the new weapon in modern Reinforcement Learning (RL) for achieving superhuman performance in practice. However, the majority of exisiting theory in reinforcement learning only applies to the setting where the agent plays against a fixed environment; it remains largely open whether self-play algorithms can be provably effective, especially when it is necessary to manage the exploration/exploitation tradeoff. We study self-play in competitive reinforcement learning under the setting of Markov games, a generalization of Markov decision processes to the two-player case. We introduce a self-play algorithm---Value Iteration with Upper/Lower Confidence Bound (VI-ULCB)---and show that it achieves regret $ ilde{\mathcal{O}}(\sqrt{T})$ after playing $T$ steps of the game, where the regret is measured by the agent's performance against a \emph{fully adversarial} opponent who can exploit the agent's strategy at \emph{any} step. We also introduce an explore-then-exploit style algorithm, which achieves a slightly worse regret of $ ilde{\mathcal{O}}(T^{2/3})$, but is guaranteed to run in polynomial time even in the worst case. To the best of our knowledge, our work presents the first line of provably sample-efficient self-play algorithms for competitive reinforcement learning.

研究动机与目标

• 为两人零和马尔可夫博弈中的竞争性强化学习提供动机并形式化表述。
• 提出在不依赖严格假设的情况下具备可证明遗憾界的自我博弈算法。
• 分析样本效率与运行时间之间的计算权衡。
• 为多智能体强化学习中的自我博弈提供PAC风格的保证和基础性结果。

提出的方法

• 通过维护乐观和悲观的Q估计(Q^up和Q^low)，将UCB概念扩展到双人设置。
• 在每个状态-行动对上对一般和型收益设定下使用纳什均衡来选择策略(μ, ν)，使其相对于(Q^up, Q^low)联合贪婪。
• 给出VI-ULCB在一般MGs中的遗憾上界，并推导出在轮次制下的专项化，具提升的遗憾和多项式运行时间。
• 引入一种计算高效的VI-Explore变体，利用无奖励探索实现多项式时间保证，遗憾为O(T^{2/3})。
• 提供从遗憾界到近均衡策略样本复杂度界的PAC风格转换。

实验结果

研究问题

• RQ1 Can self-play algorithms achieve provable, sample-efficient regret in two-player zero-sum Markov games without restrictive model assumptions?
• RQ2 How can optimistic/pessimistic value estimates be combined with general-sum Nash equilibria to guide policies in self-play?
• RQ3 What are the computational and statistical trade-offs between exact regret minimization and polynomial-time approximations in competitive RL?
• RQ4 Do turn-based Markov games admit improved regret and tractable implementations of self-play algorithms?
• RQ5 What are the fundamental lower bounds for regret and how close do the proposed methods approach them?

主要发现

Settings	Algorithm	Regret	PAC	Runtime
General Markov Game	VI-ULCB (Theorem 2)	Õ(√(H^3 S^2 A B T))	Õ(H^4 S^2 A B / ε^2)	PPAD-complete
General Markov Game	VI-explore (Theorem 5)	Õ((H^5 S^2 A B T^2)^{1/3})	Õ(H^5 S^2 A B / ε^2)	Polynomial
Mirror Descent (H=1)	(Rakhlin and Sridharan, 2013)	Õ(√(S(A+B)T))	Õ(S(A+B)/ε^2)	-
Turn-Based Markov Game	VI-ULCB (Corollary 4)	Õ(√(H^3 S^2 (A+B) T))	Õ(H^4 S^2 (A+B)/ε^2)	-
Mirror Descent (H=2)	(Theorem 10)	Õ(√(S(A+B)T))	Õ(S(A+B)/ε^2)	-
Both	Lower Bound (Corollary 7)	Ω(√(H^2 S (A+B) T))	Ω(H^2 S (A+B)/ε^2)	-

• VI-ULCB achieves a regret of Õ(√(H^3 S^2 A B T)) in general zero-sum Markov games (and Õ(√(H^3 S^2 (A+B) T)) in turn-based settings) with high-probability guarantees.
• An explore-then-exploit algorithm achieves Õ(T^{2/3}) regret with guaranteed polynomial runtime.
• A lower bound of Ω(√(H^2 S (A+B) T)) is established for (some) settings, highlighting gaps in S, A, B dependencies and motivating future work.
• The paper provides PAC bounds: with sufficient episodes K, the policy pair derived via VI-ULCB achieves near-equilibrium performance with high probability.
• In turn-based games, the Nash-equilibrium computation reduces to simpler zero-sum subproblems, enabling polynomial-time implementation and improved regret bounds.
• A PPAD-completeness caveat is discussed for the exact Nash_general_sum subroutine, motivating the polynomial-time VI-Explore variant as a practical alternative.