[Paper Review] An Optimistic Perspective on Offline Reinforcement Learning

Rishabh Agarwal, Dale Schuurmans+1 more|arXiv (Cornell University)|Jul 10, 2019

Reinforcement Learning in Robotics46 references95 citations

TL;DR

The paper shows that offline training on a large, diverse DQN replay dataset enables recent off-policy RL algorithms to outperform the best online DQN, and introduces Random Ensemble Mixture (REM) to further boost offline performance.

ABSTRACT

Off-policy reinforcement learning (RL) using a fixed offline dataset of logged interactions is an important consideration in real world applications. This paper studies offline RL using the DQN replay dataset comprising the entire replay experience of a DQN agent on 60 Atari 2600 games. We demonstrate that recent off-policy deep RL algorithms, even when trained solely on this fixed dataset, outperform the fully trained DQN agent. To enhance generalization in the offline setting, we present Random Ensemble Mixture (REM), a robust Q-learning algorithm that enforces optimal Bellman consistency on random convex combinations of multiple Q-value estimates. Offline REM trained on the DQN replay dataset surpasses strong RL baselines. Ablation studies highlight the role of offline dataset size and diversity as well as the algorithm choice in our positive results. Overall, the results here present an optimistic view that robust RL algorithms trained on sufficiently large and diverse offline datasets can lead to high quality policies. The DQN replay dataset can serve as an offline RL benchmark and is open-sourced.

Motivation & Objective

• Motivate offline RL as a practical alternative to online data collection in real-world problems.
• Evaluate whether modern off-policy RL methods can learn effectively from a fixed, large offline dataset without distribution-mismatch corrections.
• Propose robust offline learning via ensemble methods, particularly REM, to enhance generalization.
• Benchmark offline RL on Atari 2600 using the DQN Replay Dataset to establish reproducible baselines.

Proposed method

• Analyze offline RL using the DQN Replay Dataset across 60 Atari games.
• Introduce Random Ensemble Mixture (REM), a Q-learning variant that enforces optimal Bellman consistency on random convex combinations of multiple Q-value heads.
• Use a multi-head Q-network where REM forms convex combinations of head estimates to drive learning.
• Compare REM and Ensemble-DQN against offline QR-DQN, C51, and online baselines to assess offline performance.
• Provide theoretical insight (Proposition 1) on when the ensemble heads converge to the optimal Q-function.

Experimental results

Research questions

• RQ1 Can standard off-policy deep RL algorithms trained solely on offline data outperform the best policy in the offline dataset?
• RQ2 Does a robust ensemble-based approach like REM improve offline generalization and performance on Atari 2600 games?
• RQ3 How do offline dataset size and diversity affect the success of offline RL methods?
• RQ4 Is REM effective in online learning settings as well as offline settings?
• RQ5 How do different RL algorithm choices compare in the offline, large-scale Atari benchmark?

Key findings

• Offline QR-DQN and C51 improve over offline DQN but REM yields the strongest gains in the offline setting.
• With sufficient gradient updates and diverse data, offline REM surpasses the gains from fully trained online C51.
• DQN with Adam bridges the gap between QR-DQN and DQN online performance in the offline regime, while REM consistently leads in asymptotic performance.
• Offline dataset size and diversity significantly influence offline RL success, with larger and more diverse datasets enabling better generalization.
• REM’s random ensemble training acts as a robust regularizer, enabling better performance than simple ensemble methods across offline experiments.