[论文解读] Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards

Siyuan Li, Rui Wang+2 more|arXiv (Cornell University)|Oct 10, 2019

Software Engineering Research被引用 37

一句话总结

HAAR 引入基于优势的辅助奖励，以在 HRL 中同时训练高层策略和低层技能，提升学习效率和迁移性，无需任务特定奖励塑形。

ABSTRACT

Hierarchical Reinforcement Learning (HRL) is a promising approach to solving long-horizon problems with sparse and delayed rewards. Many existing HRL algorithms either use pre-trained low-level skills that are unadaptable, or require domain-specific information to define low-level rewards. In this paper, we aim to adapt low-level skills to downstream tasks while maintaining the generality of reward design. We propose an HRL framework which sets auxiliary rewards for low-level skill training based on the advantage function of the high-level policy. This auxiliary reward enables efficient, simultaneous learning of the high-level policy and low-level skills without using task-specific knowledge. In addition, we also theoretically prove that optimizing low-level skills with this auxiliary reward will increase the task return for the joint policy. Experimental results show that our algorithm dramatically outperforms other state-of-the-art HRL methods in Mujoco domains. We also find both low-level and high-level policies trained by our algorithm transferable.

研究动机与目标

• 以层级策略激励解决长时程、稀疏奖励任务。
• 使低层技能在下游任务中自适应，而不依赖领域特定奖励。
• 开发一种方法，使高层和低层策略并行训练，以提高效率和迁移性。）

提出的方法

• Two-layer HRL with a discrete high-level action selecting a low-level skill.
• Low-level policy conditioned on the high-level action that executes for k steps.
• Auxiliary low-level rewards computed from the high-level advantage function to guide skill learning.
• One-step estimation of the high-level advantage used to derive r_l via A_h(s^h_t,a^h_t) / k.
• Skill length annealing: k_i = k_1 * exp(-tau * i) to balance exploration and precision.
• Monotonic improvement proof showing joint policy improves when both levels are optimized with their respective objectives.

实验结果

研究问题

• RQ1 高层优势函数基于的辅助奖励能否以任务无关的方式引导低层技能学习？
• RQ2 高层与低层策略的并行优化是否能保持联合策略的单调改进？
• RQ3 预训练的多样化低层技能是否有利于加速高层训练和整体性能？
• RQ4 退火的长时程执行长度是否能在不牺牲最终策略质量的前提下加速学习？
• RQ5 学到的高层与低层策略是否可转移到相似任务？

主要发现

• HAAR 显著优于 Mujoco 稀疏奖励任务上的最新 HRL 方法。
• 基于高层优势的辅助奖励能提高低层技能对下游任务的适应性。
• 技能长度退火在不改变最终策略性能的前提下加速学习。
• HAAR 学到的高层与低层策略可转移到相似的下游任务。
• 该方法相比 SNN4HRL、HAC、HIRO、TRPO 等基线具有更快的收敛和更高的最终性能。）