[Paper Review] A Review for Deep Reinforcement Learning in Atari:Benchmarks, Challenges, and Solutions

Jiajun Fan|arXiv (Cornell University)|Dec 8, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

This paper critiques current evaluation metrics in Atari reinforcement learning benchmarks, arguing that human baseline scores underestimate peak human performance. It proposes a new benchmark using Human World Records (HWR) to assess both final performance and learning efficiency, revealing that even state-of-the-art agents fall short of superhuman levels across 57 games.

ABSTRACT

The Arcade Learning Environment (ALE) is proposed as an evaluation platform for empirically assessing the generality of agents across dozens of Atari 2600 games. ALE offers various challenging problems and has drawn significant attention from the deep reinforcement learning (RL) community. From Deep Q-Networks (DQN) to Agent57, RL agents seem to achieve superhuman performance in ALE. However, is this the case? In this paper, to explore this problem, we first review the current evaluation metrics in the Atari benchmarks and then reveal that the current evaluation criteria of achieving superhuman performance are inappropriate, which underestimated the human performance relative to what is possible. To handle those problems and promote the development of RL research, we propose a novel Atari benchmark based on human world records (HWR), which puts forward higher requirements for RL agents on both final performance and learning efficiency. Furthermore, we summarize the state-of-the-art (SOTA) methods in Atari benchmarks and provide benchmark results over new evaluation metrics based on human world records. We concluded that at least four open challenges hinder RL agents from achieving superhuman performance from those new benchmark results. Finally, we also discuss some promising ways to handle those problems.

Motivation & Objective

• To critique the use of human baseline scores as a proxy for human performance in Atari RL benchmarks.
• To argue that current evaluation metrics misrepresent superhuman performance due to underestimating human world records.
• To propose a new benchmark based on Human World Records (HWR) that enforces higher standards for final performance and learning efficiency.
• To evaluate state-of-the-art RL algorithms under the new HWR-based benchmark and identify persistent challenges.
• To provide a comprehensive framework for assessing superhuman capabilities in Atari environments.

Proposed method

• The paper reviews existing evaluation metrics in Atari benchmarks, particularly focusing on Human Normalized Score (HNS) and its limitations.
• It introduces a novel benchmark using Human World Records (HWR) as the performance standard, replacing the less representative human baseline.
• The authors extend the SABER evaluation system to include new metrics based on HWR, including HWRB (HWR Benchmark) and learning efficiency scores.
• Learning efficiency is quantified using inverse sample complexity, measuring how quickly agents reach high performance relative to HWR.
• The method evaluates multiple SOTA algorithms—model-free, model-based, and others—across 57 Atari games under the new HWR-based metrics.
• It introduces a multi-dimensional evaluation framework combining final performance (HWRB), median performance (MED SABER), and learning efficiency (sample efficiency) to assess agent capabilities.

Experimental results

Research questions

• RQ1 Is the current benchmark for superhuman performance in Atari RL accurate when using human baseline scores?
• RQ2 To what extent do state-of-the-art RL agents surpass human world records in final performance and learning efficiency?
• RQ3 Why do current SOTA agents fail to achieve superhuman performance despite outperforming human baselines?
• RQ4 How can learning efficiency be meaningfully measured and integrated into RL benchmarking?
• RQ5 What are the key challenges preventing RL agents from achieving true superhuman performance in Atari environments?

Key findings

• The human baseline used in most prior benchmarks significantly underestimates peak human performance, leading to misleading claims of superhuman performance.
• Even state-of-the-art agents like Agent57 and DQN-based models fail to surpass human world records in the majority of Atari 57 games.
• The proposed HWR-based benchmark (HWRB) reveals that only 22 out of 57 games were achieved by the best-performing agents, indicating a large performance gap.
• Learning efficiency remains a major bottleneck: the best agents require orders of magnitude more samples than humans to reach comparable performance levels.
• Median performance under the HWR benchmark (MED SABER) shows that agents consistently underperform relative to human world records, with median scores at 24.86% and 50.5% across different algorithm classes.
• The study identifies four open challenges hindering superhuman performance: sample efficiency, generalization across games, robustness to sparse rewards, and exploration in high-dimensional state spaces.