[Paper Review] Learning by Playing - Solving Sparse Reward Tasks from Scratch

Martin Riedmiller, Roland Hafner+7 more|arXiv (Cornell University)|Feb 28, 2018

Reinforcement Learning in RoboticsComputer Science40 references154 citations

TL;DR

Introduces Scheduled Auxiliary Control (SAC-X), a hierarchical RL framework that learns complex sparse-reward tasks from scratch by using a set of auxiliary tasks and a learned scheduler to drive exploration and policy learning.

ABSTRACT

We propose Scheduled Auxiliary Control (SAC-X), a new learning paradigm in the context of Reinforcement Learning (RL). SAC-X enables learning of complex behaviors - from scratch - in the presence of multiple sparse reward signals. To this end, the agent is equipped with a set of general auxiliary tasks, that it attempts to learn simultaneously via off-policy RL. The key idea behind our method is that active (learned) scheduling and execution of auxiliary policies allows the agent to efficiently explore its environment - enabling it to excel at sparse reward RL. Our experiments in several challenging robotic manipulation settings demonstrate the power of our approach.

Motivation & Objective

• Motivate learning in environments with sparse external rewards by leveraging auxiliary tasks. Propose a hierarchical RL framework that jointly learns auxiliary intentions and the main task. Develop a scheduler to sequence auxiliary policies to enhance exploration and external task performance. Demonstrate sample-efficient learning and transferability from simulation to real robot hardware.

Proposed method

• Define a main sparse-reward MDP augmented with a vector of auxiliary rewards and corresponding intention policies.
• Learn intentions for each auxiliary task and the main task via off-policy gradient updates with a shared replay buffer.
• Train a scheduler to sequence intentions within episodes, using a Boltzmann-inspired policy over task selections.
• Evaluate off-policy Q-functions with Retrace for stable learning across all intentions.
• Relate scheduling to curriculum learning and hierarchical RL, enabling safe combination of learned sub-policies.

Experimental results

Research questions

• RQ1 Can auxiliary, semantically grounded tasks accelerate exploration and learning in sparse-reward RL setups?
• RQ2 Does a learned scheduler for switching between auxiliary intentions improve external task performance compared to uniform or random scheduling?
• RQ3 Is SAC-X capable of solving complex robotic manipulation tasks from scratch, including stacking and cleanup, with limited external rewards?
• RQ4 How does SAC-X perform in real robot experiments in terms of learning speed and data efficiency?
• RQ5 What is the relationship between auxiliary tasks and external task performance across multiple external tasks?

Key findings

• SAC-U and SAC-Q reliably learn sparse external tasks like block stacking, outperforming DDPG which fails on these tasks.
• A learned scheduler (SAC-Q) generally accelerates learning and can achieve slightly better final performance than uniform scheduling (SAC-U).
• The approach enables learning from raw pixel inputs as well as features, with slower but still successful learning from pixels.
• Scheduling auxiliary intents significantly reduces learning time and improves reliability compared to non-scheduled baselines (IUA).
• Real-robot experiments show learning to lift a block reaches competence in about 10 hours of real-time learning, with high success on trials.
• The method scales to more complex tasks (e.g., clean-up with multiple objects) where SAC-Q learns all extrinsic tasks and auxiliary tasks, while baselines fail.