[Paper Review] Reverse Curriculum Generation for Reinforcement Learning

Carlos Florensa, David Held+3 more|arXiv (Cornell University)|Jul 17, 2017

Reinforcement Learning in RoboticsComputer Science33 references140 citations

TL;DR

The paper introduces a reverse-learning RL framework that automatically generates a curriculum of start states, expanding from a given goal to progressively harder starts, enabling efficient learning for sparse-goal tasks without demonstrations or reward shaping.

ABSTRACT

Many relevant tasks require an agent to reach a certain state, or to manipulate objects into a desired configuration. For example, we might want a robot to align and assemble a gear onto an axle or insert and turn a key in a lock. These goal-oriented tasks present a considerable challenge for reinforcement learning, since their natural reward function is sparse and prohibitive amounts of exploration are required to reach the goal and receive some learning signal. Past approaches tackle these problems by exploiting expert demonstrations or by manually designing a task-specific reward shaping function to guide the learning agent. Instead, we propose a method to learn these tasks without requiring any prior knowledge other than obtaining a single state in which the task is achieved. The robot is trained in reverse, gradually learning to reach the goal from a set of start states increasingly far from the goal. Our method automatically generates a curriculum of start states that adapts to the agent's performance, leading to efficient training on goal-oriented tasks. We demonstrate our approach on difficult simulated navigation and fine-grained manipulation problems, not solvable by state-of-the-art reinforcement learning methods.

Motivation & Objective

• Address how to learn goal-oriented tasks with sparse rewards without reward shaping or demonstrations.
• Propose a curriculum that adapts start-state distributions to the agent’s current performance.
• Develop a method to generate start states automatically by expanding from the goal using local perturbations.
• Demonstrate effectiveness on challenging robotic navigation and manipulation tasks beyond prior RL capabilities.

Proposed method

• Formalize learning with a start-state distribution that can vary across iterations to maximize learning speed.
• Define good starts as states from which the current policy achieves intermediate success.
• Generate nearby start states by short Brownian-motion-like rollouts in action space starting from seed states.
• Use a replay buffer of previously good starts to stabilize learning and enable gradual expansion.
• Iteratively train the policy with TRPO (or any on-policy method) over the adaptive start-distributions.
• Evaluate progress on the original start-state distribution to ensure generalization.

Experimental results

Research questions

• RQ1 Can adapting the start-state distribution during training accelerate learning of goal-oriented tasks with sparse rewards?
• RQ2 Does concentrating training on 'good starts' and expanding from the goal yield faster and more robust policies than uniform start-state sampling?
• RQ3 Is generating nearby states via action-space Brownian motion an effective way to grow the start-state curriculum?
• RQ4 Can the curriculum be implemented without demonstrations or reward shaping and still solve challenging manipulation tasks?

Key findings

• Adaptive start-state curricula improve learning speed and final performance compared to uniform start-state sampling.
• The method enables solving tasks not solvable by state-of-the-art RL methods at the time, including navigation and fine-grained manipulation.
• Training concentrates on good starts near the goal and expands outward, enabling backward-like learning without a model.
• Brownian-motion-based generation of nearby starts yields more efficient curriculum growth than using all previous starts.
• A simple ablation that uses all previous starts without targeting good starts underperforms the proposed approach.
• An oracle rejection-sampling upper bound suggests the approach is near practical efficiency given approximation.