[Paper Review] Behavior Regularized Offline Reinforcement Learning

Yifan Wu, George Tucker+1 more|arXiv (Cornell University)|Nov 26, 2019

Reinforcement Learning in Robotics33 references249 citations

TL;DR

The paper introduces BRAC, a flexible framework for offline RL that regularizes learned policies toward the behavior policy, showing that simple design choices can match state-of-the-art performance without heavy ensembles or adaptive regularization.

ABSTRACT

In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or otherwise yield poor performance. Accordingly, recent work has suggested a number of remedies to these issues. In this work, we introduce a general framework, behavior regularized actor critic (BRAC), to empirically evaluate recently proposed methods as well as a number of simple baselines across a variety of offline continuous control tasks. Surprisingly, we find that many of the technical complexities introduced in recent methods are unnecessary to achieve strong performance. Additional ablations provide insights into which design choices matter most in the offline RL setting.

Motivation & Objective

• Evaluate the importance of design choices in offline RL algorithms under a single framework.
• Compare behavior regularization variants (value penalty vs policy regularization) and divergence measures.
• Assess whether simple BRAC variants can match or outperform sophisticated offline RL methods.
• Provide practical guidance on which components matter most for offline RL performance.

Proposed method

• Propose the BRAC framework that unifies existing offline RL approaches through behavior regularization.
• Use value penalty or policy regularization to regularize toward the behavior policy.
• Instantiate BRAC variants with different divergences (MMD, KL, Wasserstein) and target Q-value estimation schemes.
• Perform controlled ablations on regularization type, Q-value ensembles, and divergence choice in continuous control tasks.
• Run grid searches over policy learning rates and regularization strengths to evaluate sensitivity.

Experimental results

Research questions

• RQ1 Which BRAC design choices are crucial for offline RL performance in continuous control tasks?
• RQ2 Do simple BRAC variants suffice, or are complex ensembles and adaptive regularization necessary?
• RQ3 How do different divergences for regularization compare in offline RL settings?
• RQ4 Is value penalty generally better than policy regularization in BRAC?
• RQ5 How sensitive are BRAC variants to hyperparameters and dataset quality?

Key findings

• Simple BRAC variants can achieve strong offline RL performance without heavy ensembles or adaptive regularization.
• Using a two-Q ensemble with minimum target Q-values is generally effective; larger ensembles offer limited gains.
• Value penalty tends to outperform policy regularization, though both variants beat the baseline partially trained policy.
• Different divergences (MMD, KL, Wasserstein) perform similarly in practice for regularization.
• Hyperparameter choice, especially for regularization strength, is a key factor in offline RL success.