[Paper Review] Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research

Matthias Plappert, Marcin Andrychowicz+10 more|arXiv (Cornell University)|Feb 26, 2018

Reinforcement Learning in RoboticsComputer Science17 references196 citations

TL;DR

The paper introduces a suite of challenging continuous control robotics tasks with sparse rewards under a multi-goal RL framework, and outlines concrete research ideas to improve RL algorithms, especially HER-based methods.

ABSTRACT

The purpose of this technical report is two-fold. First of all, it introduces a suite of challenging continuous control tasks (integrated with OpenAI Gym) based on currently existing robotics hardware. The tasks include pushing, sliding and pick & place with a Fetch robotic arm as well as in-hand object manipulation with a Shadow Dexterous Hand. All tasks have sparse binary rewards and follow a Multi-Goal Reinforcement Learning (RL) framework in which an agent is told what to do using an additional input. The second part of the paper presents a set of concrete research ideas for improving RL algorithms, most of which are related to Multi-Goal RL and Hindsight Experience Replay.

Motivation & Objective

• Introduce a set of challenging continuous control tasks integrated with OpenAI Gym based on Fetch and Shadow Dexterous Hand hardware.
• Adopt a multi-goal RL framework where goals are provided as additional inputs and rewards are sparse and binary.
• Evaluate baseline RL algorithms (DDPG with and without HER) on the new environments to benchmark performance.
• Propose concrete research directions to advance multi-goal RL and Hindsight Experience Replay (HER).

Proposed method

• Define and release a suite of Fetch and Shadow Hand environments with sparse binary rewards and multi-goal inputs.
• Extend OpenAI Gym with a GoalEnv-based interface featuring observation, desired_goal, and achieved_goal fields.
• Use HER to augment experience replay and evaluate under both sparse and dense reward settings.
• Benchmark DDPG with and without HER across all environments and variants, reporting median test success rates and interquartile ranges.
• Provide hyperparameter details and a reference implementation within OpenAI Baselines for reproducibility.
• research_questions_header
• research_questions_anyway
• Not used

Experimental results

Research questions

• RQ1 How can goals be generated automatically for hindsight experience replay to improve sample efficiency?
• RQ2 Can HER be made unbiased in highly stochastic environments, and how might importance sampling help?
• RQ3 What are effective ways to extend HER with hierarchical or multi-step approaches to improve learning speed and stability?
• RQ4 How can HER be integrated with on-policy RL algorithms (e.g., PPO) and combined with recent RL advances (e.g., prioritized replay, distributional RL)?
• RQ5 What strategies can improve information propagation and sample efficiency when action frequency increases in continuous control tasks?

Key findings

• DDPG with HER substantially outperforms all configurations across the majority of Fetch and Hand environments, especially with sparse rewards.
• DDPG+HER often learns well with sparse rewards, whereas vanilla DDPG struggles without HER.
• FetchReach is easy to solve for all configurations, serving as a basic sanity check for new methods.
• Hand-related tasks like HandManipulatePen remain particularly challenging, with HER not fully solving all variants.
• Dense rewards do not always yield better performance when using HER, and sparse rewards can facilitate simpler critic learning and better policy strategies.
• The paper provides a full benchmark and hyperparameter setup enabling reproducibility and comparison with existing baselines in OpenAI Baselines.
• The environments and multi-goal interface extend OpenAI Gym with a GoalEnv-compatible structure and a dictionary-based observation space, including a compute_reward function for flexible goal substitutions.