[Paper Review] Deep Q-learning from Demonstrations

Todd Hester, Matej Vecerík+12 more|arXiv (Cornell University)|Apr 12, 2017

Reinforcement Learning in Robotics307 citations

TL;DR

DQfD pre-trains on demonstrations with a combination of TD and supervised losses, then learns from both demonstration and self-generated data using prioritized replay, outperforming several baselines and achieving state-of-the-art results on multiple Atari games.

ABSTRACT

Deep reinforcement learning (RL) has achieved several high profile successes in difficult decision-making problems. However, these algorithms typically require a huge amount of data before they reach reasonable performance. In fact, their performance during learning can be extremely poor. This may be acceptable for a simulator, but it severely limits the applicability of deep RL to many real-world tasks, where the agent must learn in the real environment. In this paper we study a setting where the agent may access data from previous control of the system. We present an algorithm, Deep Q-learning from Demonstrations (DQfD), that leverages small sets of demonstration data to massively accelerate the learning process even from relatively small amounts of demonstration data and is able to automatically assess the necessary ratio of demonstration data while learning thanks to a prioritized replay mechanism. DQfD works by combining temporal difference updates with supervised classification of the demonstrator's actions. We show that DQfD has better initial performance than Prioritized Dueling Double Deep Q-Networks (PDD DQN) as it starts with better scores on the first million steps on 41 of 42 games and on average it takes PDD DQN 83 million steps to catch up to DQfD's performance. DQfD learns to out-perform the best demonstration given in 14 of 42 games. In addition, DQfD leverages human demonstrations to achieve state-of-the-art results for 11 games. Finally, we show that DQfD performs better than three related algorithms for incorporating demonstration data into DQN.

Motivation & Objective

• Motivate learning in real-world RL settings where accurate simulators are unavailable but demonstration data exists.
• Pre-train a deep Q-network on demonstration data to learn a value function that satisfies the Bellman equation.
• Enable online learning by continuing to train with a mix of demonstration and self-generated data via prioritized replay.
• Demonstrate that incorporating demonstrations yields faster initial performance and state-of-the-art results on several Atari games.

Proposed method

• Pre-train on demonstration data using four losses: 1-step double Q-learning, n-step double Q-learning, a large-margin supervised classification loss, and L2 regularization.
• Ground the demonstrator's actions with a large-margin loss to push other actions below the demonstrator's action by a margin.
• Incorporate n-step TD losses to propagate values from the demonstration trajectory.
• Use prioritized experience replay with demonstration transitions receiving a priority bonus to balance demo vs. agent data.
• During online learning, maintain demonstration data in the replay buffer permanently and sample from a mix of demonstration and self-generated data.

Experimental results

Research questions

• RQ1 Can a deep Q-network be effectively pre-trained from demonstrations and then continue learning from its own experience?
• RQ2 Does combining TD updates with supervised imitation losses improve early performance and overall learning when demonstrations are available?
• RQ3 How should demonstration data and self-generated data be balanced during online learning to maximize performance?
• RQ4 Do demonstration-augmented Deep Q-networks achieve superior performance on hard exploration Atari games compared to non-demonstration baselines?

Key findings

• DQfD achieves better initial performance than Prioritized Dueling Double DQN (PDD DQN) on 41 of 42 games in the first million steps.
• On average, PDD DQN requires about 83 million steps to catch up to DQfD's performance.
• DQfD learns to outperform the best demonstration in 14 of 42 games and achieves state-of-the-art results for 11 games.
• DQfD outperforms three related demonstration-augmented algorithms and pure imitation learning across multiple metrics.
• DQfD provides strong gains even with a relatively small demonstration set (5,574 to 75,472 transitions per game).
• The combination of all four losses during pre-training is shown to be critical for stability and performance.