[Paper Review] Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewards

Siyuan Li, Rui Wang+2 more|arXiv (Cornell University)|Oct 10, 2019

Software Engineering Research37 citations

TL;DR

HAAR introduces advantage-based auxiliary rewards to train high-level policies and low-level skills concurrently in HRL, improving learning efficiency and transferability without task-specific reward shaping.

ABSTRACT

Hierarchical Reinforcement Learning (HRL) is a promising approach to solving long-horizon problems with sparse and delayed rewards. Many existing HRL algorithms either use pre-trained low-level skills that are unadaptable, or require domain-specific information to define low-level rewards. In this paper, we aim to adapt low-level skills to downstream tasks while maintaining the generality of reward design. We propose an HRL framework which sets auxiliary rewards for low-level skill training based on the advantage function of the high-level policy. This auxiliary reward enables efficient, simultaneous learning of the high-level policy and low-level skills without using task-specific knowledge. In addition, we also theoretically prove that optimizing low-level skills with this auxiliary reward will increase the task return for the joint policy. Experimental results show that our algorithm dramatically outperforms other state-of-the-art HRL methods in Mujoco domains. We also find both low-level and high-level policies trained by our algorithm transferable.

Motivation & Objective

• Motivate solving long-horizon, sparse-reward tasks with hierarchical policies.
• Enable adaptation of low-level skills to downstream tasks without relying on domain-specific rewards.
• Develop a method to train both high-level and low-level policies concurrently for better efficiency and transferability.

Proposed method

• Two-layer HRL with a discrete high-level action selecting a low-level skill.
• Low-level policy conditioned on the high-level action that executes for k steps.
• Auxiliary low-level rewards computed from the high-level advantage function to guide skill learning.
• One-step estimation of the high-level advantage used to derive r_l via A_h(s^h_t,a^h_t) / k.
• Skill length annealing: k_i = k_1 * exp(-tau * i) to balance exploration and precision.
• Monotonic improvement proof showing joint policy improves when both levels are optimized with their respective objectives.

Experimental results

Research questions

• RQ1 Can auxiliary rewards based on the high-level advantage function guide low-level skill learning in a task-agnostic way?
• RQ2 Does concurrent optimization of high- and low-level policies preserve monotonic improvement of the joint policy?
• RQ3 Are pre-trained diverse low-level skills beneficial for accelerating high-level training and overall performance?
• RQ4 Do annealed long-horizon execution lengths accelerate learning without sacrificing final policy quality?
• RQ5 Are the learned high- and low-level policies transferable to similar tasks?

Key findings

• HAAR significantly outperforms state-of-the-art HRL methods on Mujoco sparse-reward tasks.
• Auxiliary rewards based on the high-level advantage improve low-level skill adaptation to downstream tasks.
• Skill length annealing accelerates learning without changing final policy performance.
• Both high-level and low-level policies trained by HAAR are transferable to similar downstream tasks.
• The approach yields faster convergence and higher final performance than baselines like SNN4HRL, HAC, HIRO, and TRPO.