[Paper Review] Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning

Silviu Pitis, Harris Chan+3 more|arXiv (Cornell University)|Jul 6, 2020

Reinforcement Learning in Robotics38 citations

TL;DR

The paper introduces MEGA and OMEGA, intrinsic goal selection methods that maximize entropy of the historical achieved goal distribution to enable efficient exploration in long-horizon multi-goal RL, achieving substantial sample efficiency improvements over prior methods.

ABSTRACT

What goals should a multi-goal reinforcement learning agent pursue during training in long-horizon tasks? When the desired (test time) goal distribution is too distant to offer a useful learning signal, we argue that the agent should not pursue unobtainable goals. Instead, it should set its own intrinsic goals that maximize the entropy of the historical achieved goal distribution. We propose to optimize this objective by having the agent pursue past achieved goals in sparsely explored areas of the goal space, which focuses exploration on the frontier of the achievable goal set. We show that our strategy achieves an order of magnitude better sample efficiency than the prior state of the art on long-horizon multi-goal tasks including maze navigation and block stacking.

Motivation & Objective

• Motivate the difficulty of learning long-horizon multi-goal tasks with sparse rewards.
• Propose intrinsic goal selection that maximizes entropy of past achieved goals to drive exploration.
• Introduce an annealed objective (OMEGA) that blends unsupervised MEGA with the supervised goal distribution.
• Show how density-based, frontier-focused exploration improves sample efficiency in long-horizon tasks.

Proposed method

• Define MEGA as maximizing the entropy of the historical achieved goal distribution to expand exploration when pdg and pag do not overlap.
• Formulate OMEGA as an annealed objective that transitions from MEGA to the original distribution-matching objective using a mixture with a uniform over current support.
• Propose a practical SELECT mechanism (MEGA SELECT) that chooses behavioural goals from low-density regions of pag using a learned density model.
• Present a minimum-density heuristic as a tractable approximation to the oracle objective, sampling candidate goals from the buffer and selecting the least dense one.
• Integrate with hindsight experience replay and off-policy learning (DDPG) to achieve efficiency.
• Discuss connections to empowerment and relate to prior intrinsic motivation methods such as DISCERN, RIG, and Skew-Fit.

Experimental results

Research questions

• RQ1 How can intrinsic goal setting be formulated to address the long-horizon exploration problem in multi-goal RL?
• RQ2 Can maximizing entropy of the past achieved goal distribution accelerate learning for sparse-reward, long-horizon tasks?
• RQ3 Does an annealed MEGA/OMEGA objective improve sample efficiency relative to existing goal-relabelling baselines?
• RQ4 What practical SELECT strategies best approximate optimal entropy gain in continuous, high-dimensional goal spaces?

Key findings

• MEGA-focused exploration increases the entropy of the achieved goal distribution faster than baselines.
• OMEGA effectively transitions from intrinsic MEGA exploration to pursuing the desired goal distribution, maintaining stability and efficiency.
• MEGA/OMEGA solve long-horizon tasks (maze navigation and block stacking) with substantially higher sample efficiency than prior state-of-the-art baselines.
• Minimum-density goal selection outperforms diverse or achieved-goal sampling on long-horizon tasks.
• Empirical results show MEGA/OMEGA reach desired goal regions far faster than previous methods, e.g., solving tasks where prior approaches struggle.