[论文解读] No-Regret Learning Dynamics for Extensive-Form Correlated Equilibrium

Andrea Celli, Alberto Marchesi+2 more|arXiv (Cornell University)|Apr 1, 2020

Advanced Bandit Algorithms Research参考文献 32被引用 16

一句话总结

本文提出了ICFR，这是首个在具有完美记忆的通用和n人序列形式博弈中收敛至广义形式相关均衡（EFCE）的非耦合无遗憾学习动态。通过定义触发遗憾并将其在每个信息集处局部分解，该算法实现了高效、去中心化的学习，以高概率渐近收敛至EFCE，将Hart和Mas-Colell在正常形式博弈中的结果推广至序列性、不完美信息博弈。

ABSTRACT

The existence of simple, uncoupled no-regret dynamics that converge to correlated equilibria in normal-form games is a celebrated result in the theory of multi-agent systems. Specifically, it has been known for more than 20 years that when all players seek to minimize their internal regret in a repeated normal-form game, the empirical frequency of play converges to a normal-form correlated equilibrium. Extensive-form (that is, tree-form) games generalize normal-form games by modeling both sequential and simultaneous moves, as well as private information. Because of the sequential nature and presence of partial information in the game, extensive-form correlation has significantly different properties than the normal-form counterpart, many of which are still open research directions. Extensive-form correlated equilibrium (EFCE) has been proposed as the natural extensive-form counterpart to normal-form correlated equilibrium. However, it was currently unknown whether EFCE emerges as the result of uncoupled agent dynamics. In this paper, we give the first uncoupled no-regret dynamics that converge to the set of EFCEs in $n$-player general-sum extensive-form games with perfect recall. First, we introduce a notion of trigger regret in extensive-form games, which extends that of internal regret in normal-form games. When each player has low trigger regret, the empirical frequency of play is close to an EFCE. Then, we give an efficient no-trigger-regret algorithm. Our algorithm decomposes trigger regret into local subproblems at each decision point for the player, and constructs a global strategy of the player from the local solutions at each decision point.

研究动机与目标

• 弥合对非耦合无遗憾学习动态是否能在通用和序列形式博弈中收敛至EFCE的理解差距。
• 将经典无遗憾收敛结果从正常形式博弈推广至更复杂的序列性、不完美信息博弈场景。
• 设计一种高效、去中心化的学习算法，最小化触发遗憾，并在无需协调或全局知识的情况下确保收敛至EFCE。
• 为EFCE提供有限时间内的高概率收敛边界，强化先前几乎必然收敛的结果。

提出的方法

• 为序列形式博弈引入一种新型触发遗憾概念，推广自正常形式博弈中的内部遗憾。
• 将触发遗憾分解为每个玩家在信息集处的局部子问题，以支持分布式计算。
• 通过聚合每个信息集处遗憾最小化器的局部解，构建全局策略。
• 在每个信息集处结合使用内部与外部遗憾最小化器，以控制触发遗憾。
• 协调学习过程，使得每轮仅使用一个遗憾最小化器即可维持全局收敛。
• 将算法置于φ-遗憾最小化框架中，统一并强化理论保证。

实验结果

研究问题

• RQ1 在具有完美记忆的n人通用和序列形式博弈中，非耦合无遗憾学习动态能否收敛至EFCE？
• RQ2 如何将内部遗憾推广至序列性、不完美信息的序列形式博弈设定？
• RQ3 是否可能设计一种去中心化、高效的算法，以最小化触发遗憾并确保收敛至EFCE？
• RQ4 在该类动态下，EFCE的有限时间收敛保证为何？

主要发现

• ICFR确保了在极限情况下，实际对弈频率以几乎必然收敛至EFCE。
• 该算法为EFCE提供了高概率的有限时间收敛边界，相较于会议版本的渐近保证有显著改进。
• 在包含超过9,000个信息集、两注限注的复杂Leduc扑克实例上，ICFR在9小时内即达到ε-EFCE，优于以往方法在24小时内均未能达到ε=0.1的表现。
• 在EFCE、EFCCE与NFCCE之间，偏离激励的收敛速率相似，表明ICFR在各类相关均衡解概念下均具鲁棒性。
• 该方法将Hart和Mas-Colell的无遗憾动态推广至序列情形，建立了学习动态与序列形式博弈中相关均衡之间的基础性联系。