[Paper Review] Rainbow: Combining Improvements in Deep Reinforcement Learning

Matteo Hessel, Joseph Modayil+8 more|arXiv (Cornell University)|Oct 6, 2017

Evolutionary Algorithms and Applications427 citations

TL;DR

Rainbow combines six extensions to DQN (Double Q-learning, Prioritized Replay, Dueling Networks, Multi-step learning, Distributional RL, Noisy Nets) and achieves state-of-the-art data efficiency and final performance on 57 Atari games.

ABSTRACT

The deep reinforcement learning community has made several independent improvements to the DQN algorithm. However, it is unclear which of these extensions are complementary and can be fruitfully combined. This paper examines six extensions to the DQN algorithm and empirically studies their combination. Our experiments show that the combination provides state-of-the-art performance on the Atari 2600 benchmark, both in terms of data efficiency and final performance. We also provide results from a detailed ablation study that shows the contribution of each component to overall performance.

Motivation & Objective

• Motivate the need to understand which DQN improvements are complementary.
• Investigate whether integrating multiple extensions yields additive performance gains.
• Quantify data efficiency and final performance of Rainbow on a broad Atari benchmark.
• Provide ablations to identify the contribution of each component.

Proposed method

• Integrate six DQN extensions into a single agent (Double Q-learning, Prioritized Replay, Dueling Networks, Multi-step targets, Distributional Q-learning, Noisy Nets).
• Replace 1-step distributional loss with a multi-step distributional loss and use a dueling architecture for return distributions.
• Prioritize transitions by KL loss instead of absolute TD error to guide replay.
• Use a multi-step target in the distributional Bellman update with a fixed atom distribution.
• Employ Noisy Nets with factorised Gaussian noise for exploration.
• Evaluate on 57 Atari 2600 games with standard normalization and two testing regimes; perform ablations on each component.

Experimental results

Research questions

• RQ1 Do the six DQN extensions complement each other when combined into a single agent?
• RQ2 What is the contribution of each component to Rainbow's performance across games and learning phases?
• RQ3 How does Rainbow compare to existing baselines in data efficiency and final performance on the Atari suite?

Key findings

• Rainbow achieves state-of-the-art performance on 57 Atari games in terms of data efficiency and final performance.
• Rainbow matches DQN's final performance after 7M frames and surpasses other baselines within 44M frames.
• Rainbow achieves a median human-normalized score of 223% in the no-ops regime and 153% in the human-starts regime when fully trained.
• Ablation studies show prioritized replay and multi-step learning are the most crucial components for performance gains.
• Noisy Nets generally improve median performance; distributional learning remains beneficial, especially in higher-performing games.