[Paper Review] Reinforcement Learning with Unsupervised Auxiliary Tasks

Max Jaderberg, Volodymyr Mnih+5 more|arXiv (Cornell University)|Nov 16, 2016

Reinforcement Learning in Robotics271 citations

TL;DR

UNREAL augments A3C with unsupervised auxiliary tasks (pixel control, reward prediction, value function replay) to learn richer representations, boosting data efficiency and performance on Atari and Labyrinth.

ABSTRACT

Deep reinforcement learning agents have achieved state-of-the-art results by directly maximising cumulative reward. However, environments contain a much wider variety of possible training signals. In this paper, we introduce an agent that also maximises many other pseudo-reward functions simultaneously by reinforcement learning. All of these tasks share a common representation that, like unsupervised learning, continues to develop in the absence of extrinsic rewards. We also introduce a novel mechanism for focusing this representation upon extrinsic rewards, so that learning can rapidly adapt to the most relevant aspects of the actual task. Our agent significantly outperforms the previous state-of-the-art on Atari, averaging 880\% expert human performance, and a challenging suite of first-person, three-dimensional \emph{Labyrinth} tasks leading to a mean speedup in learning of 10$ imes$ and averaging 87\% expert human performance on Labyrinth.

Motivation & Objective

• Promote faster and more robust learning by learning from auxiliary pseudo-rewards in addition to the extrinsic reward.
• Develop a shared representation (CNN-LSTM) that benefits both the base task and auxiliary tasks.
• Focus the learned representation toward extrinsic reward through reward-focused auxiliary objectives.
• Leverage experience replay to accelerate value iteration and support off-policy auxiliary learning.

Proposed method

• Introduce auxiliary control tasks (pixel control and feature control) as additional pseudo-reward functions to train separate auxiliary policies sharing parameters with the base agent.
• Add auxiliary reward task (reward prediction) to bias feature learning toward predicting imminent rewards without biasing policy.
• Incorporate value function replay to perform off-policy value updates on replayed data.
• Use skewed replay sampling to oversample rewarding events for reward prediction and maintain stability with off-policy auxiliary tasks.
• Combine losses into UNREAL objective: L_UNREAL = L_A3C + lambda_VR L_VR + lambda_PC sum_c L_Q^(c) + lambda_RP L_RP (Equation 2).
• Share CNN-LSTM representation across base and auxiliary tasks and apply on-policy A3C updates with off-policy auxiliary task updates.

Experimental results

Research questions

• RQ1 Does adding auxiliary control and reward tasks improve sample efficiency over vanilla A3C in challenging visual environments?
• RQ2 Are auxiliary tasks more robust to hyperparameters (learning rate, entropy cost) than the baseline?
• RQ3 Do auxiliary tasks improve final performance on 3D Labyrinth and Atari suites beyond standard unsupervised reconstructions?
• RQ4 How does sharing a common representation across tasks affect learning speed and final policy quality?

Key findings

• UNREAL yields 87% human-normalized score on Labyrinth (vs 54% for A3C).
• UNREAL achieves a mean 10x speedup in learning on Labyrinth (up to 18x on some levels).
• On Atari, UNREAL surpasses prior state-of-the-art with 880% mean and 250% median human-normalized performance.
• UNREAL shows greater robustness to hyperparameter settings than A3C across tasks.
• Auxiliary tasks, especially pixel control and reward prediction, significantly improve data efficiency and final performance compared to baselines and reconstruction-based approaches.