[논문 리뷰] Data-Efficient Hierarchical Reinforcement Learning

Ofir Nachum, Shixiang Gu외 2명|arXiv (Cornell University)|2018. 05. 21.

Reinforcement Learning in Robotics참고 문헌 3인용 수 265

한 줄 요약

HIRO를 제안하는 두-layer HRL 에이전트로, 오프폴리시 보정으로 오프폴리시 학습을 수행하며 표본 효율이 높고 로봇의 보행 및 물체 상호작용 과제에서 강력한 성능을 달성합니다.

ABSTRACT

Hierarchical reinforcement learning (HRL) is a promising approach to extend traditional reinforcement learning (RL) methods to solve more complex tasks. Yet, the majority of current HRL methods require careful task-specific design and on-policy training, making them difficult to apply in real-world scenarios. In this paper, we study how we can develop HRL algorithms that are general, in that they do not make onerous additional assumptions beyond standard RL algorithms, and efficient, in the sense that they can be used with modest numbers of interaction samples, making them suitable for real-world problems such as robotic control. For generality, we develop a scheme where lower-level controllers are supervised with goals that are learned and proposed automatically by the higher-level controllers. To address efficiency, we propose to use off-policy experience for both higher and lower-level training. This poses a considerable challenge, since changes to the lower-level behaviors change the action space for the higher-level policy, and we introduce an off-policy correction to remedy this challenge. This allows us to take advantage of recent advances in off-policy model-free RL to learn both higher- and lower-level policies using substantially fewer environment interactions than on-policy algorithms. We term the resulting HRL agent HIRO and find that it is generally applicable and highly sample-efficient. Our experiments show that HIRO can be used to learn highly complex behaviors for simulated robots, such as pushing objects and utilizing them to reach target locations, learning from only a few million samples, equivalent to a few days of real-time interaction. In comparisons with a number of prior HRL methods, we find that our approach substantially outperforms previous state-of-the-art techniques.

연구 동기 및 목표

• 일반적이고 데이터 효율적인 HRL을 표준 RL 구성요소와 함께 작동하도록 동기를 부여하고 개발한다.
• 높은 수준의 컨트롤러가 제안하는 목표에 의해 안내되는 하위 정책을 학습한다.
• 샘플 효율성을 개선하기 위해 계층의 두 레벨 모두에서 오프폴리시 학습을 가능하게 한다.
• 하위 정책의 비정상성(non-stationarity)을 해결하기 위한 오프폴리시 보정을 도입한다.
• 제한된 상호작용 데이터로도 시뮬레이션 로봇 작업에서 강력한 성능을 입증한다.]
• method2Lesson
• Two-layer hierarchy with a high-level policy (goals) and a low-level policy (actions).
• Lower level receives a goal g_t and incurs intrinsic reward r = -||s_t + g_t - s_{t+1}||_2; higher level optimizes over temporally-extended goals every c steps.
• Higher level experiences are relabeled (off-policy correction) to maximize the probability that past low-level actions would occur under the current lower-level controller, enabling off-policy learning.
• Both policies are trained with off-policy TD methods (TD3) using replay buffers.
• Goals are defined directly in raw state observations, avoiding learned embeddings or manual goal spaces.
• An eight-candidate relabeling procedure plus original and difference-based goals are used for high-level relabeling to approximate argmax of likelihood.

제안 방법

• Two-layer hierarchy with a high-level policy (goals) and a low-level policy (actions).
• Lower level receives a goal g_t and incurs intrinsic reward r = -||s_t + g_t - s_{t+1}||_2; higher level optimizes over temporally-extended goals every c steps.
• Higher level experiences are relabeled (off-policy correction) to maximize the probability that past low-level actions would occur under the current lower-level controller, enabling off-policy learning.
• Both policies are trained with off-policy TD methods (TD3) using replay buffers.
• Goals are defined directly in raw state observations, avoiding learned embeddings or manual goal spaces.
• An eight-candidate relabeling procedure plus original and difference-based goals are used for high-level relabeling to approximate argmax of likelihood.

실험 결과

연구 질문

• RQ1 제안된 오프폴리시 보정을 갖춘 두 수준의 HRL 시스템이 복잡한 과제를 효율적으로 학습할 수 있는가?
• RQ2 하위 정책의 목표로 원시 상태 관찰을 사용하는 것이 학습 속도와 성능을 향상시키는가?
• RQ3 제안된 오프폴리시 보정이 Naive 오프폴리시 HRL에 비해 안정성 및 표본 효율성에 어떤 영향을 미치는가?
• RQ4 HIRO의 도전적 로봇 위치추적 및 물체상호작용 과제에서의 성능은 기존 HRL 방법과 어떻게 비교되는가?

주요 결과

개미 수집	개미 미로	개미 밀기	개미 낙하
HIRO	3.02 ± 1.49	0.99 ± 0.01	0.92 ± 0.04	0.66 ± 0.07
FuN representation	0.03 ± 0.01	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0
FuN transition PG	0.41 ± 0.06	0.0 ± 0.0	0.56 ± 0.39	0.01 ± 0.02
FuN cos similarity	0.85 ± 1.17	0.16 ± 0.33	0.06 ± 0.17	0.07 ± 0.22
FuN	0.01 ± 0.01	0.0 ± 0.0	0.0 ± 0.0	0.0 ± 0.0
SNN4HRL	1.92 ± 0.52	0.0 ± 0.0	0.02 ± 0.01	0.0 ± 0.0
VIME	1.42 ± 0.90	0.0 ± 0.0	0.02 ± 0.02	0.0 ± 0.0

• HIRO는 Ant Gather, Ant Maze, Ant Push, Ant Fall 과제에서 강력한 성능을 달성한다.
• 10M 스텝에서 HIRO는 FuN 변형, SNN4HRL, VIME을 포함한 기반 방법들보다 모든 과제에서 우수한 성능을 보이며, Ant Gather가 하위 레벨 사전학습으로 가장 근접한 경쟁자이다.
• HIRO는 빠른 학습을 보여주며 환경 스텝 몇 백만 회 후에 복잡한 과제를 해결한다(현실 세계 상호작용 며칠에 해당하는 수준).
• 오프폴리시 보정은 더 어려운 과제에서 안정성과 성능에 결정적이며, Naive 오프폴리시 학습은 Ant Push 및 Ant Fall에서 악화된다.
• 원시 상태 관찰을 목표로 사용하는 것은 즉각적인 내재적 보상 신호와 과제 간 간단한 일반화를 제공한다.