[Paper Review] Offline Reinforcement Learning Hands-On

Louis Monier, Jakub Kmec+5 more|arXiv (Cornell University)|Nov 29, 2020

Reinforcement Learning in Robotics12 references4 citations

TL;DR

This paper provides a hands-on, empirical investigation of offline reinforcement learning methods—behavioral cloning (BC), CRR, and CQL—on synthetic datasets with varying quality (random, medium, expert) across discrete and continuous control environments. It finds that BC remains a robust, strong baseline, especially on continuous control tasks, while CQL and CRR show mixed performance depending on data quality and environment complexity, with CQL being difficult to tune and stabilize despite strong theoretical grounding.

ABSTRACT

Offline Reinforcement Learning (RL) aims to turn large datasets into powerful decision-making engines without any online interactions with the environment. This great promise has motivated a large amount of research that hopes to replicate the success RL has experienced in simulation settings. This work ambitions to reflect upon these efforts from a practitioner viewpoint. We start by discussing the dataset properties that we hypothesise can characterise the type of offline methods that will be the most successful. We then verify these claims through a set of experiments and designed datasets generated from environments with both discrete and continuous action spaces. We experimentally validate that diversity and high-return examples in the data are crucial to the success of offline RL and show that behavioural cloning remains a strong contender compared to its contemporaries. Overall, this work stands as a tutorial to help people build their intuition on today's offline RL methods and their applicability.

Motivation & Objective

• To evaluate the practical performance of leading offline RL algorithms—BC, CRR, and CQL—on controlled, synthetic datasets with varying data quality.
• To investigate how dataset properties such as diversity and return distribution affect the success of offline RL methods.
• To provide a hands-on, empirically grounded comparison to help practitioners select appropriate algorithms based on data quality and task characteristics.
• To highlight practical challenges in implementing state-of-the-art methods like CQL, including hyperparameter sensitivity and implementation discrepancies not addressed in the original papers.

Proposed method

• The authors generate synthetic datasets using policies of varying quality—random, medium, and expert—across environments with both discrete and continuous action spaces.
• They evaluate three offline RL methods: behavioral cloning (BC), Critic Regularized Regression (CRR), and Conservative Q-Learning (CQL), using standard deep RL architectures.
• For CRR, they use both mean and max advantage estimation with a binary function to avoid numerical instability from exponential filtering.
• For CQL, they implement the recommended LogSumExp approximation and include additional sampling (N≈50–100) and temperature scaling to improve regularizer accuracy, based on author feedback.
• They conduct ablation studies to assess the impact of data quality and algorithmic choices, including hyperparameter tuning and implementation fidelity.
• They validate results on both simple (e.g., PointMaze-v0) and complex environments to test generalization and robustness.

Experimental results

Research questions

• RQ1 How does the quality of the offline dataset (random, medium, expert) affect the performance of BC, CRR, and CQL?
• RQ2 Can CQL and CRR outperform the behavior policy that generated the dataset, especially on medium-quality data?
• RQ3 Why do CQL and CRR show inconsistent performance across different environments and data quality levels?
• RQ4 To what extent is behavioral cloning a reliable baseline in continuous control settings with high-quality data?
• RQ5 What practical challenges arise in implementing CQL, and how do they affect reproducibility and performance?

Key findings

• Behavioral cloning (BC) consistently performs well across all datasets and environments, especially in continuous control tasks, and remains a strong, robust baseline.
• CQL achieves strong performance on medium-quality data in discrete action spaces and can outperform the behavior policy, but shows high instability and hyperparameter sensitivity in practice.
• CRR occasionally outperforms the behavior policy on medium-quality data in discrete settings but fails to generalize on complex continuous control tasks.
• In the PointMaze-v0 environment, both BC and CQL successfully recover expert behavior, while CRR gets stuck in a local minimum, indicating poor generalization.
• CQL is difficult to implement correctly due to undocumented implementation details—such as the use of N≈50–100 samples and temperature scaling—leading to high variance and failure to reproduce results without careful tuning.
• The study reveals that no single algorithm uniformly outperforms others; success depends critically on data quality and environment complexity, with high-quality data being essential for strong offline RL performance.