[Paper Review] A Minimalist Approach to Offline Reinforcement Learning

Scott Fujimoto, Shixiang Gu|arXiv (Cornell University)|Jun 12, 2021

Reinforcement Learning in Robotics56 references164 citations

TL;DR

TD3+BC shows that adding a single behavior cloning term to TD3 and normalizing data matches state-of-the-art offline RL performance with substantially less complexity and compute.

ABSTRACT

Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data. Due to errors in value estimation from out-of-distribution actions, most offline RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make an RL algorithm work offline comes at the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such as generative models, while adjusting the underlying RL algorithm. In this paper we aim to make a deep RL algorithm work while making minimal changes. We find that we can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is a simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overhead of previous methods.

Motivation & Objective

• Motivate a minimalist offline RL approach that reduces implementation and hyperparameter overhead.
• Investigate whether simple changes to an online algorithm can perform well offline without extra components.
• Show that data normalization and a BC term can stabilize and improve offline learning.
• Provide an easily reproducible baseline aligning with state-of-the-art performance on standard benchmarks.

Proposed method

• Start from TD3 and add a behavior cloning regularization term to the policy update.
• Normalize dataset state features to have zero mean and unit variance.
• Introduce a lambda scaling for the BC/QL balance: lambda = alpha / (1/N) sum|Q(s,a)| (estimated per minibatch).
• Use a single hyperparameter alpha (default 2.5) to control the strength of the regularizer.
• Keep changes minimal: only a few lines of code beyond the base TD3 update.

Experimental results

Research questions

• RQ1 Can a deep RL algorithm be made effective offline with minimal changes to the base online algorithm?
• RQ2 Does a simple BC regularizer plus data normalization suffice to match state-of-the-art offline RL methods on standard benchmarks?
• RQ3 What is the impact of the normalization and the BC term on stability and performance in offline RL?

Key findings

• TD3+BC achieves comparable performance to Fisher-BRC on the D4RL MuJoCo benchmarks.
• TD3+BC requires significantly less computation time than CQL and Fisher-BRC (approximately less than half the total training time).
• State normalization provides a non-trivial stability and performance benefit in offline RL.
• A single hyperparameter (alpha) governs the balance between RL and imitation, with robustness across tasks in many settings.