[Paper Review] Off-Policy Deep Reinforcement Learning without Exploration

Scott Fujimoto, David Meger+1 more|arXiv (Cornell University)|Dec 7, 2018

Reinforcement Learning in Robotics70 references279 citations

TL;DR

The paper identifies extrapolation error as a barrier to learning from fixed batch data with off-policy methods and introduces BCQ, a batch-constrained DRL algorithm that learns effectively without interacting with the environment by generating actions similar to the batch and selecting high-value ones.

ABSTRACT

Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off-policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off-policy algorithms, batch-constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on-policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.

Motivation & Objective

• Motivate batch reinforcement learning as a practical setting where data collection is costly or risky.
• Characterize extrapolation error as a core obstacle to true off-policy learning from fixed data.
• Propose batch-constrained reinforcement learning to align the learned policy with the data distribution.
• Introduce BCQ to learn from arbitrary fixed batch data in continuous control tasks.

Proposed method

• Define batch-constrained policies to minimize mismatch between policy actions and batch data.
• Develop BCQ combining a state-conditioned generative model with a Q-network to select actions resembling the batch.
• Use a perturbation model to diversify candidate actions within a bounded range.
• Adopt Clipped Double Q-learning with a weighted minimum to penalize uncertain future states.
• Provide theoretical results showing extrapolation error can be eliminated in deterministic MDPs under batch coherence.
• Offer a practical deep RL implementation (BCQ) with a VAE-based generator and action perturbations.

Experimental results

Research questions

• RQ1 Can off-policy learning from fixed batch data be made reliable by restricting the policy to the batch distribution?
• RQ2 How can we jointly learn a generative model of batch actions and a value function to maximize returns without exploring?
• RQ3 Does a batch-constrained approach improve stability and performance in continuous control tasks compared to standard off-policy methods?
• RQ4 Under what conditions can batch-constrained learning guarantee unbiased value estimation or policy improvement within the batch?
• RQ5 How does BCQ perform with imperfect or expert demonstrations compared to traditional imitation and RL baselines?

Key findings

• BCQ matches or outperforms behavioral policies across multiple MuJoCo tasks in batch settings.
• The BCQ value estimates remain stable, unlike DDPG and DQN which diverge under batch learning.
• BCQ effectively leverages expert and suboptimal batch data without further environment interaction.
• In imperfect demonstrations, BCQ outperforms both deep RL and imitation baselines by disentangling poor from expert actions.
• A single set of hyper-parameters suffices across tasks, demonstrating robustness of BCQ.