[Paper Review] Automatic Goal Generation for Reinforcement Learning Agents

Carlos Florensa, David Held+2 more|arXiv (Cornell University)|May 17, 2017

Reinforcement Learning in RoboticsComputer Science47 references144 citations

TL;DR

The paper introduces Goal GAN, an adversarial framework that automatically generates intermediate-difficulty goals to train a single policy to reach a diverse, continuous set of goals with sparse rewards, achieving an automatic curriculum and improved sample efficiency.

ABSTRACT

Reinforcement learning is a powerful technique to train an agent to perform a task. However, an agent that is trained using reinforcement learning is only capable of achieving the single task that is specified via its reward function. Such an approach does not scale well to settings in which an agent needs to perform a diverse set of tasks, such as navigating to varying positions in a room or moving objects to varying locations. Instead, we propose a method that allows an agent to automatically discover the range of tasks that it is capable of performing. We use a generator network to propose tasks for the agent to try to achieve, specified as goal states. The generator network is optimized using adversarial training to produce tasks that are always at the appropriate level of difficulty for the agent. Our method thus automatically produces a curriculum of tasks for the agent to learn. We show that, by using this framework, an agent can efficiently and automatically learn to perform a wide set of tasks without requiring any prior knowledge of its environment. Our method can also learn to achieve tasks with sparse rewards, which traditionally pose significant challenges.

Motivation & Objective

• Motivate learning a policy that can reach a diverse, continuous set of goals rather than a single task.
• Enable automatic curriculum generation to match the agent's current capabilities.
• Develop a goal-conditioned RL framework that works with sparse rewards without hand-engineered rewards.
• Demonstrate improved sample efficiency and scalability to higher-dimensional goal spaces.

Proposed method

• Define goals as parameterized subsets of the state space with binary rewards for reaching a goal.
• Introduce Goal GAN to generate goals that lie in the Goals of Intermediate Difficulty (GOID) for the current policy.
• Label goals by the agent’s observed success to train the GAN with positive and negative examples.
• Iteratively train the policy on GOID samples and update the GAN based on policy performance.
• Use TRPO with GAE as the underlying RL optimizer for policy updates.

Experimental results

Research questions

• RQ1 Can automatic curriculum generation via a Goal GAN improve the sample efficiency of learning to reach multiple goals compared to baselines?
• RQ2 Does the Goal GAN adaptively sample goals of intermediate difficulty and track multimodal goal distributions?
• RQ3 How does the method scale to higher-dimensional or more complex goal spaces while maintaining performance?
• RQ4 Is the approach robust to sparse reward settings without hand-designed rewards?
• RQ5 Can the method prevent forgetting while expanding the set of reachable goals over time?

Key findings

• Goal GAN accelerates learning by focusing on goals with intermediate difficulty, outperforming uniform sampling and several baselines.
• The generator dynamically shifts to GOID, producing goals that are neither too easy nor too hard as the policy improves.
• The method tracks multimodal goal distributions and maintains diverse goal coverage, including in maze-like environments.
• In higher-dimensional goal spaces, the approach remains effective by generating goals within the feasible subset, avoiding uninformative samples.
• A rejection-sampling oracle variant confirms that the GOID-based sampling is near-optimal, while the full GAN-based method remains far more sample-efficient.