[论文解读] Agent57: Outperforming the Atari Human Benchmark

Adrià Puigdomènech Badia, Bilal Piot+5 more|arXiv (Cornell University)|Mar 30, 2020

Reinforcement Learning in Robotics参考文献 47被引用 143

一句话总结

Agent57 是第一位在所有 57 个 Atari 游戏中超越人类基准的深度强化学习代理，通过引入元控制器并将状态-动作值参数分离，在探索、开发利用以及长期信用分配之间进行自适应平衡。

ABSTRACT

Atari games have been a long-standing benchmark in the reinforcement learning (RL) community for the past decade. This benchmark was proposed to test general competency of RL algorithms. Previous work has achieved good average performance by doing outstandingly well on many games of the set, but very poorly in several of the most challenging games. We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games. To achieve this result, we train a neural network which parameterizes a family of policies ranging from very exploratory to purely exploitative. We propose an adaptive mechanism to choose which policy to prioritize throughout the training process. Additionally, we utilize a novel parameterization of the architecture that allows for more consistent and stable learning.

研究动机与目标

• 展示一个通用的 Atari 57 代理，在所有游戏中击败人类基准。
• 相较 NGU 改善探索与长期信用分配，达到更均衡的表现。
• 引入稳定学习的架构与训练机制，使其在多样化游戏中保持稳定。
• 展示自适应策略选择和更长的反向传播时间对学习稳定性与最终表现的提升。

提出的方法

• Parameterize Q-values into extrinsic and intrinsic components: Q(x,a,j;θ)=Q(x,a,j;θ^e)+β_j Q(x,a,j;θ^i).
• Train two separate networks for intrinsic and extrinsic values with transformed Retrace losses.
• Introduce a meta-controller (per actor) using a non-stationary multi-armed bandit to adapt policy selection (β_j, γ_j) over episodes.
• Use a distributed RL setup with a central prioritized replay buffer and multiple actors.
• Employ a longer backpropagation-through-time window (e.g., 160 vs 80) to improve long-term credit assignment.
• Evaluate on all 57 Atari games with CHNS/ HNS metrics and compare to baselines (R2D2, NGU, MuZero).

实验结果

研究问题

• RQ1 Can a single agent achieve over 100% human-normalized score across all 57 Atari games?
• RQ2 Does separating intrinsic and extrinsic value functions improve training stability and performance across diverse games?
• RQ3 Can a meta-controller adaptively select exploration/exploitation policies to enhance generality and tail performance?
• RQ4 Does increasing backpropagation through time window improve long-term credit assignment without sacrificing overall performance?

主要发现

• Agent57 achieves 100% capped human-normalized score across the 57 games.
• Agent57 surpasses the human benchmark on all 57 games, with higher tail performance than several strong baselines.
• The separate intrinsic/extrinsic value networks improve robustness to intrinsic reward scaling and boost performance on hard-exploration games.
• Adaptive exploration via a meta-controller yields substantial gains in CHNS for both NGU and R2D2 baselines.
• Longer backpropagation-through-time windows improve training stability and final performance, especially on Solaris.
• On the challenging 10-game subset, each proposed improvement contributes to the final 100% CHNS, indicating necessity of all components.