[Paper Review] Benchmarking Batch Deep Reinforcement Learning Algorithms

Scott Fujimoto, Edoardo Conti+2 more|arXiv (Cornell University)|Oct 3, 2019

Reinforcement Learning in Robotics42 references160 citations

TL;DR

The paper benchmarks off-policy and batch DRL algorithms in a fixed Atari batch setting and introduces a discrete-action BCQ variant that outperforms prior methods, often matching or exceeding the behavior policy.

ABSTRACT

Widely-used deep reinforcement learning algorithms have been shown to fail in the batch setting--learning from a fixed data set without interaction with the environment. Following this result, there have been several papers showing reasonable performances under a variety of environments and batch settings. In this paper, we benchmark the performance of recent off-policy and batch reinforcement learning algorithms under unified settings on the Atari domain, with data generated by a single partially-trained behavioral policy. We find that under these conditions, many of these algorithms underperform DQN trained online with the same amount of data, as well as the partially-trained behavioral policy. To introduce a strong baseline, we adapt the Batch-Constrained Q-learning algorithm to a discrete-action setting, and show it outperforms all existing algorithms at this task.

Motivation & Objective

• Assess how current off-policy and batch DRL algorithms perform under a unified Atari batch setting.
• Evaluate extrapolation error and stability across discrete-action environments.
• Identify a strong, simple baseline for discrete batch DRL in fixed data scenarios.

Proposed method

• Review and implement several batch DRL algorithms (QR-DQN, REM, BCQ, KL-Control, SPIBB-DQN) in a unified Atari setting with a single 10M-transition batch.
• Adapt BCQ to discrete actions to serve as a strong baseline.
• Diagnose extrapolation error through value estimates and stability analysis across games.
• Compare against online DQN and the batch-derived behavioral policy across 9 Atari games.

Experimental results

Research questions

• RQ1 Do standard off-policy DRL methods perform well in a single-behavioral-policy batch setting on Atari?
• RQ2 Can batch/constrained methods like BCQ provide robust performance in discrete-action batch RL?
• RQ3 How does extrapolation error manifest in discrete batch RL, and does a distributional or constrained approach mitigate it?
• RQ4 What is the relative performance of discrete-action BCQ versus existing batch RL algorithms in this setting?

Key findings

• Standard off-policy DRL algorithms underperform online DQN and the behavioral policy in the single-behavioral-policy batch setup.
• QR-DQN often outperforms DQN but generally still underperforms the noisy behavioral policy.
• Batch RL methods like BCQ outperform other methods and often match or exceed the noiseless behavioral policy.
• KL-Control shows strong initial performance but is not robust across games, with value divergence causing failures in several cases.
• The discrete-action BCQ variant achieves state-of-the-art results among tested batch DRL methods in this setting.
• Stable value estimates correlate with better batch learning performance, highlighting extrapolation error mitigation as crucial.