[论文解读] A Game Theoretic Framework for Model Based Reinforcement Learning

Aravind Rajeswaran, Igor Mordatch+1 more|arXiv (Cornell University)|Apr 16, 2020

Reinforcement Learning in Robotics参考文献 67被引用 43

一句话总结

本文将基于模型的强化学习建模为策略玩家和模型玩家之间的两人博弈，并使用基于Stackelberg博弈的算法（PAL和MAL）求解，该算法在样本效率方面表现出色，且能够扩展到高维任务。

ABSTRACT

Model-based reinforcement learning (MBRL) has recently gained immense interest due to its potential for sample efficiency and ability to incorporate off-policy data. However, designing stable and efficient MBRL algorithms using rich function approximators have remained challenging. To help expose the practical challenges in MBRL and simplify algorithm design from the lens of abstraction, we develop a new framework that casts MBRL as a game between: (1) a policy player, which attempts to maximize rewards under the learned model; (2) a model player, which attempts to fit the real-world data collected by the policy player. For algorithm development, we construct a Stackelberg game between the two players, and show that it can be solved with approximate bi-level optimization. This gives rise to two natural families of algorithms for MBRL based on which player is chosen as the leader in the Stackelberg game. Together, they encapsulate, unify, and generalize many previous MBRL algorithms. Furthermore, our framework is consistent with and provides a clear basis for heuristics known to be important in practice from prior works. Finally, through experiments we validate that our proposed algorithms are highly sample efficient, match the asymptotic performance of model-free policy gradient, and scale gracefully to high-dimensional tasks like dexterous hand manipulation. Additional details and code can be obtained from the project page at https://sites.google.com/view/mbrl-game

研究动机与目标

• 揭示基于模型的强化学习的实际挑战，并通过抽象统一算法设计。
• 将MBRL表述为策略优化与世界模型拟合之间的两人博弈。
• 开发基于Stackelberg的算法以在连续博弈中计算均衡。
• 展示在样本效率和对高维任务的可扩展性方面的改进。
• 提供将先前的MBRL方法联系起来并推广的见解。

提出的方法

• 将基于模型的RL公式化为两人博弈：策略玩家在所学模型中最大化奖励，模型玩家在策略引导的状态分布下最小化预测误差。
• 采用Stackelberg博弈结构以实现稳定的双层优化并推导出实用的基于梯度的更新。
• 引入两种领导者-跟随者变体：Policy as Leader (PAL) 和 Model as Leader (MAL)，各自具有具体的嵌套优化方案。
• 使用一阶近似来求解双层更新，能够迭代地先更新模型再更新策略（PAL）或先更新策略再更新模型（MAL）。
• 用神经网络表示策略和动力学模型，并使用集成和熵正则化来提高鲁棒性。

实验结果

研究问题

• RQ1 将MBRL视为两人博弈能否产生稳定高效的学习动力学？
• RQ2 基于Stackelberg的PAL和MAL算法是否在样本效率和可扩展性方面优于先前的MBRL和无模型方法？
• RQ3 在动力学或目标分布变化的环境中，PAL和MAL的比较如何？
• RQ4 有哪些理论保证将均衡质量与环境中的策略最优性联系起来？

主要发现

• PAL和MAL在连续控制任务中表现出稳定、接近单调的学习。
• PAL和MAL在样本效率方面优于先前的基于模型和无模型方法，并能扩展到高维的 dexterous 任务。
• PAL往往比MAL学得更快，而MAL在应对目标分布变化方面更占优势。
• BR (Best Response) 导致不稳定，而 GDA (Gradient Descent-Ascent) 在某些情形下速度慢或不稳定。
• 在所评估的任务中，这些方法的渐近性能与无模型策略梯度基线相当。
• 通过一个有原则的博弈论视角将保守性与数据聚合联系起来，统一并推广了先前的MBRL方法。