[Paper Review] Observe and Look Further: Achieving Consistent Performance on Atari

Tobias Pohlen, Bilal Piot+11 more|arXiv (Cornell University)|May 29, 2018

Reinforcement Learning in RoboticsComputer Science18 references85 citations

TL;DR

The paper introduces Ape-X DQfD, a distributed DQN variant with a transformed Bellman operator, temporal consistency loss, and demonstrations, achieving human-level performance on 40/42 Atari games and solving Montezuma’s Revenge first level.

ABSTRACT

Despite significant advances in the field of deep Reinforcement Learning (RL), today's algorithms still fail to learn human-level policies consistently over a set of diverse tasks such as Atari 2600 games. We identify three key challenges that any algorithm needs to master in order to perform well on all games: processing diverse reward distributions, reasoning over long time horizons, and exploring efficiently. In this paper, we propose an algorithm that addresses each of these challenges and is able to learn human-level policies on nearly all Atari games. A new transformed Bellman operator allows our algorithm to process rewards of varying densities and scales; an auxiliary temporal consistency loss allows us to train stably using a discount factor of $γ= 0.999$ (instead of $γ= 0.99$) extending the effective planning horizon by an order of magnitude; and we ease the exploration problem by using human demonstrations that guide the agent towards rewarding states. When tested on a set of 42 Atari games, our algorithm exceeds the performance of an average human on 40 games using a common set of hyper parameters. Furthermore, it is the first deep RL algorithm to solve the first level of Montezuma's Revenge.

Motivation & Objective

• Identify key challenges to achieving human-level performance across diverse Atari games (reward distributions, long-horizon reasoning, exploration).
• Develop a stable learning algorithm that can handle unclipped rewards and high discount factors without changing optimal policies.
• Leverage expert demonstrations within a distributed RL framework to improve exploration and sample efficiency.
• Demonstrate performance gains over prior DQN variants across a large Atari suite, including sparse-reward games.

Proposed method

• Introduce a transformed Bellman operator to reduce target variance without clipping rewards.
• Utilize an auxiliary temporal consistency (TC) loss to enable stable learning with a high discount factor (gamma=0.999).
• Combine Ape-X distributed experience replay with Deep Q-learning from Demonstrations (DQfD) to fuse online agent data with expert demonstrations.
• Apply an imitation loss only on the best expert trajectory while maintaining a fixed actor–expert data mix during training.
• Provide an ablation study to quantify contributions of the transformed operator, TC loss, and demonstrations.

Experimental results

Research questions

• RQ1 Can a transformed Bellman operator stabilize Q-learning across diverse reward scales without clipping?
• RQ2 Does a temporal consistency loss enable stable learning and effective planning horizons with gamma close to 1?
• RQ3 How does incorporating demonstrations within a distributed DQN framework affect performance and exploration on Atari games?
• RQ4 To what extent does the proposed method improve performance on sparse-reward games like Montezuma’s Revenge and Pitfall!?

Key findings

• The algorithm exceeds average human performance on 40 of 42 Atari games with the same hyperparameters.
• It is the first deep RL method to complete the first level of Montezuma’s Revenge.
• Using a higher discount factor (gamma=0.999) with TC loss yields extended planning horizons and stable learning.
• The combination of transformed Bellman operator, TC loss, and demonstrations yields improved consistency and performance across games compared to baselines.
• Deeper network architecture with gamma=0.999 further improves results, achieving 40/42 games above average human.