[Paper Review] Hindsight Experience Replay

Marcin Andrychowicz, Filip Wolski+8 more|arXiv (Cornell University)|Jul 5, 2017

Reinforcement Learning in Robotics44 references352 citations

TL;DR

HER enables sample-efficient learning from sparse binary rewards by replaying each episode with alternative goals, improving off-policy RL across multi-goal tasks for robotics.

ABSTRACT

Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task.

Motivation & Objective

• Motivate the difficulty of reward shaping in robotics and the need for learning from sparse signals.
• Introduce a method to learn universal policies that incorporate goals as inputs.
• Show that replaying experiences with altered goals can dramatically improve learning efficiency.
• Demonstrate that policies trained in simulation can transfer to a physical robot.

Proposed method

• Use universal value function approximators that take state and goal as input.
• Replay each episode with the original goal and with additional goals, such as the final state achieved in the episode (or other strategies).
• Apply off-policy RL algorithms (e.g., DQN, DDPG, NAF, SDQN) with a replay buffer augmented by hindsight transitions.
• Formulate rewards as sparse binary or explored through different strategies for replay goals.
• Provide an algorithm description (Algorithm 1) for integrating HER with off-policy RL.
• Analyze how different goal-sampling strategies affect learning (e.g., final, future, episode, random).

Experimental results

Research questions

• RQ1 Can off-policy RL with hindsight replay learn effectively from sparse, binary rewards?
• RQ2 Does replaying trajectories with alternative goals enable learning for multi-goal manipulation tasks?
• RQ3 Which strategies for selecting additional goals for replay maximize learning efficiency?
• RQ4 Does HER enable transfer from simulation to a physical robot without fine-tuning?

Key findings

• DDPG with HER solves pushing, sliding, and pick-and-place tasks where standard RL fails.
• HER remains effective with sparse rewards and can outperform reward-shaped alternatives in the tested tasks.
• Using future/episode/partial future goals for replay yields better performance, especially for the sliding task.
• Policies trained in simulation can deploy on a physical Fetch robot with no fine-tuning after adding observation noise during retraining.
• Reward shaping did not improve performance for these tasks under the tested forms.
• Training with multiple goals speeds learning even when only one goal is of interest.