[Paper Review] Maximum Entropy-Regularized Multi-Goal Reinforcement Learning

Rui Zhao, Xudong Sun+1 more|arXiv (Cornell University)|May 21, 2019

Reinforcement Learning in Robotics47 citations

TL;DR

Introduces a reward-weighted entropy objective for multi-goal RL and a maximum entropy-based prioritization (MEP) to learn from diverse achieved goals, improving performance and sample-efficiency in multi-goal robotic tasks.

ABSTRACT

In Multi-Goal Reinforcement Learning, an agent learns to achieve multiple goals with a goal-conditioned policy. During learning, the agent first collects the trajectories into a replay buffer, and later these trajectories are selected randomly for replay. However, the achieved goals in the replay buffer are often biased towards the behavior policies. From a Bayesian perspective, when there is no prior knowledge about the target goal distribution, the agent should learn uniformly from diverse achieved goals. Therefore, we first propose a novel multi-goal RL objective based on weighted entropy. This objective encourages the agent to maximize the expected return, as well as to achieve more diverse goals. Secondly, we developed a maximum entropy-based prioritization framework to optimize the proposed objective. For evaluation of this framework, we combine it with Deep Deterministic Policy Gradient, both with or without Hindsight Experience Replay. On a set of multi-goal robotic tasks of OpenAI Gym, we compare our method with other baselines and show promising improvements in both performance and sample-efficiency.

Motivation & Objective

• Motivate learning from diverse achieved goals when target goal distributions are unknown.
• Combine maximum entropy with multi-goal RL to reduce replay bias toward behavior policies.
• Derive a safe surrogate objective that lower-bounds the original entropy-regularized objective.
• Provide a practical optimization framework (MEP) that maximizes policy return while increasing goal diversity.

Proposed method

• Define a weighted entropy objective for multi-goal RL that weights trajectories by accumulated reward.
• Derive a safe surrogate objective as a lower bound to stabilize optimization.
• Introduce a maximum entropy-based prioritization (MEP) framework using a density model of goal trajectories.
• Model p(tau^g) with a Latent Variable Model (Mixture of Gaussians) and use complementary density to construct the proposal distribution q(tau^g).
• Optimize using off-policy methods (DDPG with or without HER) plus the MEP prioritization to replay diverse goals.
• Provide an algorithm (MEP) and demonstrate improvements on OpenAI Gym multi-goal robotic tasks.

Experimental results

Research questions

• RQ1 Does incorporating a goal-entropy term via MEP improve off-policy multi-goal RL methods (DDPG, DDPG+HER)?
• RQ2 Does MEP enhance sample efficiency and performance in robotic manipulation tasks?
• RQ3 How does MEP affect the entropy of the achieved goal distribution during training?

Key findings

• MEP accelerates convergence and improves final performance across six robotic tasks compared to baselines.
• MEP achieves faster training and up to 39.34 percentage points performance gain in some tasks.
• MEP reduces training time relative to prior PER while delivering stronger performance.
• MEP improves sample efficiency by approximately a factor of two on average across environments.
• Entropy of the achieved goal distribution increases during MEP training, confirming the intended effect.
• In arm tasks, DDPG+MEP is about 1.2x the baseline training time, while DDPG+PER is about 5x, highlighting computational efficiency advantages of MEP.