[论文解读] Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning

Silviu Pitis, Harris Chan+3 more|arXiv (Cornell University)|Jul 6, 2020

Reinforcement Learning in Robotics被引用 38

一句话总结

论文提出 MEGA 和 OMEGA，是最大化历史实现目标分布熵的内在目标选择方法，以在长时序多目标强化学习中实现高效探索，相较于先前方法在样本效率上获得显著提升。

ABSTRACT

What goals should a multi-goal reinforcement learning agent pursue during training in long-horizon tasks? When the desired (test time) goal distribution is too distant to offer a useful learning signal, we argue that the agent should not pursue unobtainable goals. Instead, it should set its own intrinsic goals that maximize the entropy of the historical achieved goal distribution. We propose to optimize this objective by having the agent pursue past achieved goals in sparsely explored areas of the goal space, which focuses exploration on the frontier of the achievable goal set. We show that our strategy achieves an order of magnitude better sample efficiency than the prior state of the art on long-horizon multi-goal tasks including maze navigation and block stacking.

研究动机与目标

• 激发在稀疏奖励下学习长时序多目标任务的困难。
• 提出通过最大化过去实现目标的熵来驱动探索的内在目标选择。
• 引入一个退火目标（OMEGA），将无监督的 MEGA 与有监督的目标分布相结合。
• 展示基于密度、前沿聚焦的探索如何在长时序任务中提升样本效率。

提出的方法

• 将 MEGA 定义为最大化历史实现目标分布的熵，以在 pdg 和 pag 不重叠时扩展探索。
• 将 OMEGA 表述为一个退火目标，通过对当前支撑上的均匀分布进行混合，使其从 MEGA 过渡到原始分布匹配目标。
• 提出一个实用的 SELECT 机制（MEGA SELECT），使用学习到的密度模型从 pag 的低密度区域选择行为目标。
• 给出一个最小密度启发式作为对 oracle 目标的可行近似，从缓冲区采样候选目标并选择密度最低的一个。
• 与后见经验回放和离策略学习（DDPG）相结合以提高效率。
• 讨论与赋能（empowerment）的联系，并将其与先前的内在动机方法如 DISCERN、RIG 和 Skew-Fit 联系起来。

实验结果

研究问题

• RQ1 如何将内在目标设定表述为解决多目标强化学习中长时序探索的问题？
• RQ2 最大化过去实现目标分布的熵是否能加快稀疏奖励、长时序任务的学习？
• RQ3 相较于现有的目标重新标签基线，退火式的 MEGA/OMEGA 目标是否能提高样本效率？
• RQ4 在连续高维目标空间中，哪些实用的 SELECT 策略能最好地逼近最优熵增？

主要发现

• MEGA-focused exploration increases the entropy of the achieved goal distribution faster than baselines.
• OMEGA effectively transitions from intrinsic MEGA exploration to pursuing the desired goal distribution, maintaining stability and efficiency.
• MEGA/OMEGA solve long-horizon tasks (maze navigation and block stacking) with substantially higher sample efficiency than prior state-of-the-art baselines.
• Minimum-density goal selection outperforms diverse or achieved-goal sampling on long-horizon tasks.
• Empirical results show MEGA/OMEGA reach desired goal regions far faster than previous methods, e.g., solving tasks where prior approaches struggle.